club-3090
Recipes for serving LLMs locally on RTX 3090s (and other NVIDIA cards): working compose configs,
patches and measured numbers, across vLLM, SGLang, llama.cpp (mainline, our moe-cache fork and PrismML's
ternary build), ExLlamaV3, and engines built for one model: Strata, Project Maya, bucko-vllm and
HyperQwen (docs/engines/). Pick a model for your card count, launch it with
one command, and get an OpenAI-compatible API.
📣 The newest slugs, numbers and caveats are posted in Announcements first. The repo docs catch up afterwards, so when a page here and an announcement disagree, the announcement is newer. The card pages below link each slug to its announcement thread.
Pick your path
Every measured result lives in BENCHMARKS.md. Terms like TPS, KV cache and MTP are
explained in the GLOSSARY; engines in
INFERENCE_ENGINES.md; quant names in
QUANTIZATION.md. The full docs index is docs/README.md.
Quick start
Needs Linux (on Windows, set up WSL2 first), Docker with the NVIDIA Container
Toolkit, and room on disk for the weights.
git clone https://github.com/noonghunna/club-3090.git && cd club-3090
bash scripts/setup.sh # pick a model; downloads and SHA-verifies the weights
bash scripts/launch.sh # pick a config for your GPUs; boots it and runs a quick check
# the launcher prints a test request for what you booted; for the Qwen3.6-27B dual default:
curl -sf http://localhost:8010/v1/chat/completions -H "Content-Type: application/json" \
-d '{"model":"qwen3.6-27b","messages":[{"role":"user","content":"Capital of France?"}],"max_tokens":200}'
bash scripts/switch.sh --list # every config this machine can run (--all adds retired ones)
bash scripts/switch.sh <slug> # switch to another one (🧪 slugs need --force)
bash scripts/update.sh # later: pull the latest stack and re-run setup
bash scripts/settings.sh show # your saved settings (model dir, default pins, …) and where each comes from
Settings you save (the weights dir, your default config) live in ~/.config/club-3090/, shared by every
checkout: where settings live.
Upgrading from before v0.12.0? What moved, and how to move yours.
Prefer a screen to the CLI? c3 is a terminal cockpit for the same flow: browse the catalog,
serve with one key, watch GPUs and containers, run health checks. Install it with
uv pip install -e tools/serve-cockpit and run c3; details in
tools/serve-cockpit/.
Other hardware
The configs are hardware-class aware, and contributors have run them on 4090s, 5090s, A-series
cards and more: Can I use a 4090? ·
Can I use a 5090? · HARDWARE.md (card
classes, power limits, NVLink, PCIe).
More than chat
Club 3090 AI Studio adds open-weight image, video and audio
generation alongside a chat model, driven from Open WebUI. The image bundle is one command:
bash scripts/setup-image-studio.sh.
Where things live
models/<model>/<engine>/compose/<topology>/<quant>/<serving>.yml one compose per slug
models/<model>/<engine>/patches/ vendored fixes for that engine
scripts/ setup, launch, switch, verify, bench, quality, report, update
docs/ the guides linked above
tools/ the c3 cockpit, kv-calc, chart and docs generators
BENCHMARKS.md every measured row · CHANGELOG.md what changed, dated
Engines and hardware are documented once, under docs/; everything specific to a model (quants,
quirks, recipes) sits under models/<name>/.
Community
- 💬 Discord — casual chat, hardware questions, share what you're running. Use for synchronous Q&A.
- 📋 GitHub Discussions — async, searchable. Best for cross-rig benchmark drops, "should I tune X" type threads, and anything you want others to find via search.
- 🐛 GitHub Issues — bug reports, regression repros, concrete asks. Triage ladder in FAQ before filing.
Community projects
Projects in the club-3090 ecosystem maintained outside this repo:
- VykosX/club-3090-server — single-file installer adding a server-management layer on top of club-3090: browser admin panel on
:8008/admin, OpenAI-compatible reverse proxy on :8009 with multi-backend routing, GPU-aware multi-instance orchestration, fan/power controls, audit logs, and per-user API auth/quota. Headless Arch + Debian/Ubuntu friendly. Started 2026-05-05, AGPL-3.0; see discussion #108 for the announcement and current WIP status. Not yet officially adopted — listed here as a community pointer until it converges on a stable surface area.
If you've built something that integrates with club-3090 and you'd like a pointer added here, open a discussion.
Credits
The stack stands on a lot of shoulders:
- Qwen team (@Alibaba_Qwen) — for the base models and the MTP head architecture
- Lorbus — for the AutoRound INT4 quant with preserved BF16
mtp.fc (the model this whole stack runs on)
- Sandermage — root-caused vllm#40880 (MTP × TurboQuant cudagraph capture) and shipped the fix that made spec-decode work on consumer Ampere. The patch tree it shipped in is no longer used here, but the diagnosis stands and is upstream at vllm#40914.
- vibhavagarwal5 — TurboQuant landing PR + tracking issue #40069
- vLLM project — the engine + active maintenance
- llama.cpp — the alternative engine path
- The engine authors whose work the catalog serves as it is: Niko1221 (Strata), mw00 (Project Maya, a Strata fork for GLM-5.3-Flash), DominikBucko (bucko-vllm), syv-ai (HyperQwen), turboderp (ExLlamaV3, served by theroyallab's TabbyAPI) and PrismML (ternary llama.cpp). We pin their images, or build one from their release where they publish none; the engines are theirs.
License
Apache 2.0. Do what you want with it. If you get better numbers on your rig — open an issue. If you add a new model with working configs — open a PR.
