1Cat-LLM
Make Volta Fast Again
Modern LLM inference for NVIDIA Tesla V100 / SM70
1Cat-vLLM is becoming 1Cat-LLM.
Broader support. More efficient kernels. More models to choose from.
We started by making modern models fast on V100. The next chapter carries that work further: broader model and runtime support, more efficient execution, and more choice in how models are packaged and quantized.
New GGUF support brings a wider choice of community-quantized checkpoints for supported architectures, with more options for model size, memory use and quality. The new name reflects that direction, with Volta engineering at its foundation.
The rename is planned. Repository links, release filenames and installation commands below retain their current names during the transition.
Recommended checkpoints:
Target: QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4
MoE: RadixArk/Qwen3.8-Flash-Next-NVFP4
DFlash2 draft: incoai/Qwen3.8-27B-DFlash2
4× Tesla V100 16GB · Qwen3.8-27B-NVFP4 + DFlash2 · ≈260 tok/s
Tesla V100 was released in 2017.
Its Tensor Cores did not suddenly become useless.
The software stack simply stopped being optimized seriously for SM70.
1Cat-LLM grows out of 1Cat-vLLM, a vLLM engineering fork that treats NVIDIA Volta / SM70 / Tesla V100 as a first-class optimization target.
We are not satisfied with:
“The latest model can start on V100.”
Our goal is:
Make modern models actually run fast on V100.
The recorded four-card Tesla V100 16GB demo runs Qwen3.8-27B-NVFP4 + DFlash2 through 1Cat-vLLM at roughly:
≈260 tokens/s
Demo: 4× V100 running Qwen3.8-27B-NVFP4-DFlash2
≈260 tok/s is a real-machine demo headline, not a universal fixed decode rate.
Every benchmark below retains its own hardware, model, context length, batch size, KV dtype, sampling policy, and speculative-decoding contract. Attention TFLOP/s, prefill tok/s, target-only decode tok/s, and speculative decode tok/s are not interchangeable metrics.
Latest published release: v1.5.1, October 3, 2026. The release measurements below use 4× V100-SXM2 32GB; they are a separate contract from the 16GB demo.
This README also covers merged development work through main b379775d, October 5, 2026. The newer GGUF and mixed-load changes require a corresponding source/development build; they are not included merely by installing the v1.5.1 release wheel.
📊 Performance First
SM70 Flash-V100 resolves --kv-cache-dtype fp8 to E4M3; the 27B release profile explicitly selects fp8_e4m3. DFlash2 verification retains FP32 attention state and FP32 logits on its admitted paths.
Keep the Python package and Flash-V100 extension from the same build. The original E4M3 repair requires precision revision 4; current multi-head and request-major routes require newer native capabilities. See the precision contract and concurrent long-context validation.
Historical E5M2/FP16-partial performance results below keep their original
configuration and are not speed claims for these precision defaults.
v1.5.1: Measured Release-to-Release Decode
| Input / workload | v1.5.0 | v1.5.1 | Comparison |
|---|
| 4K · single request | 103.87 tok/s | 229.34 tok/s | +120.80% |
| 32K · single request | 49.04 tok/s | 201.18 tok/s | +310.24% |
| 128K · single request | No completed baseline | 171.43 tok/s | No speedup calculated |
| 32K · C4 common decode window | Not measured under the same contract | 445.45 aggregate tok/s | New-version result only |
Speedup percentages are calculated from the displayed throughput values.
Single-request contract: official wheels, the same 4× V100-SXM2-32GB / TP4, QUASAR 27B NVFP4 target, DFlash2 q7, FP16 compute/draft KV, E4M3 target KV, CUDA 12.8 / Torch 2.10, prefix caching, asynchronous scheduling and CUDA Graphs. Both versions use the 1.5.1 launch settings: 262144 maximum context, 8192-token prefill budget, four sequence slots, memory utilization 0.80, KV/Mamba blocks 2048/8192. Sampling is T=1, top-p=.95, top-k=20, seed=0, up to 256 output tokens with normal EOS. Each value is the median of three warm requests after one cold request, from one successful startup per version.
Pure decode excludes TTFT/prefill. At 32K, the old wheel reused 28,672 prefix tokens while the new wheel recomputed the prompt; both decoded with the same 32,768-token input context. This is a comparison under one shared deployment recipe, not each version's best possible tuning. The old 128K request did not finish within the 180-second test timeout.
C4 contract: fixed 32K synthetic inputs, 256 forced output tokens per request, T=.7 / top-p=.8 / top-k=20, fixed per-request seeds, one warmup cohort and three measured cohorts from one startup. Throughput counts returned token IDs only while all four requests are decoding. Forced-length timing is separate from natural-EOS quality testing.
Evidence and complete settings: v1.5.1 release notes. These release checks include output-health spot checks, not a new broad dataset-quality campaign.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
