HyperQwen

One request against stock vLLM, replayed in real time from recorded token arrivals, then 64 at once. 1080p60 MP4 · how it is made

Today: Qwen3.8-27B on a single 24 GB consumer GPU with vLLM — 150k token context and an OpenAI-compatible API with key auth, in two ready-made modes. On an RTX 3090 at 250 W: 127 tok/s single-stream in single-user mode, ~1,035 tok/s aggregate decode at 64 concurrent in batch mode, and 381 tok/s when the answer quotes its own prompt.

Where it is going: more Qwen models, more classes of card. What this repo really is, under the serving setup, is a patch series against a pinned vLLM plus a model-preparation pipeline — requantized heads, a calibrated draft vocabulary, speculation that drafts out of the prompt. Most of that is not specific to one checkpoint or one card; it is just that one checkpoint and one card are what has been measured to death. See Roadmap for what is and is not portable yet, and Field tests wanted if you have half an hour on a GPU we have never touched.

Quick start

git clone https://github.com/syv-ai/HyperQwen && cd HyperQwen
cp .env.example .env                      # PowerShell: Copy-Item .env.example .env

docker compose --profile single up -d     # one or a few people chatting
docker compose --profile batch  up -d     # API backend, many concurrent requests

First start pulls the image (9.5 GB) and requantizes the model (~20 GB, once, into ./models), then serves on :18020. One GPU runs one mode at a time. While the model downloads, compose shows prepare-1 Waiting and the server container stays at Starting; on a slow link that is tens of minutes, not a hang. Follow it with docker compose logs -f prepare (then make logs for the server). The model needs one card with 24 GB or more, or two cards together: on 16 GB or smaller cards, see docs/multi-gpu.md before the first start.

• Before exposing it — compose publishes the port on 127.0.0.1 only (BIND in .env, default 127.0.0.1), so another machine cannot reach it until you set a key and BIND=0.0.0.0; outside a container, no key means the server binds 127.0.0.1 only: echo "VLLM_API_KEY=$(openssl rand -hex 24)" >> .env && echo BIND=0.0.0.0 >> .env
• Linux — Docker CE with the NVIDIA Container Toolkit, not the snap Docker; sudo docker run --rm --gpus all ubuntu nvidia-smi should list your cards first (docs/docker.md). No Docker: docs/install.md
• Docker Desktop on WSL2 — keep VLLM_WSL2_ENABLE_PIN_MEMORY=1 in .env, or the V2 runner aborts with RuntimeError: UVA is not available
• No compose, or no Docker at all — docs/docker.md · docs/install.md

Which setup do I want?

Three questions, and the answers are the whole of your .env.

flowchart TD
  Q1{"Who is sending<br/>the requests?"}
  Q1 -->|"an API backend, a pipeline,<br/>8+ at once"| A["<b>A — batch</b>"]
  Q1 -->|"one or a few<br/>people chatting"| Q2{"Longest prompt<br/>you will send?"}
  Q2 -->|"under 64k tokens"| Q3{"Do the answers quote<br/>the prompt back?"}
  Q2 -->|"up to 150k"| D["<b>D — long context</b>"]
  Q2 -->|"200k and beyond"| E["<b>E — huge context</b>"]
  Q3 -->|"no — normal chat"| B["<b>B — single, default</b>"]
  Q3 -->|"yes — code edits, RAG,<br/>rewrites, translation"| C["<b>C — reproduction</b>"]

Starting from the .env you copied in Quick start, which already ships B:

	change in .env	start with	what you get
A	nothing	--profile batch	~1,035 tok/s aggregate at 64 concurrent
B	nothing	--profile single	127 tok/s single stream, 64k context
C	add DFLASH_TOKENS=15	--profile single	381 tok/s while quoting, at 4 slots and 56k
D	SPEC=mtp and CTX=long	--profile single	150k context, ~95-100 tok/s
E	add CTX=huge	--profile single	240k context, 67 tok/s mixed and 164 while quoting

Torn between B and C: B is the safe default, and C only pays off when the output really does repeat the input — it trades request slots and context for it. D drops back to MTP speculation on purpose, because DFlash2 past 64k is worth it only for reproduction and loses to SPEC=mtp CTX=long about 2:1 on everything else; at E the KVarN cache buys the context instead, so DFlash2 stays on. A needs no edit at all — batch ignores SPEC and says so on startup. Every knob, and what each costs: single-user/README.md · batch/README.md.

On a compute capability 8.6 GPU (RTX 3090-class), SPEC=mtp CTX=long and SPEC=dflash2 CTX=long run on an int8 KV cache read by the pth-flash-attn kernel by default (PTH_FA=1, source vendored in pthfa/): the target and the drafter on --attention-backend CUSTOM --kv-cache-dtype int8_per_token_head. That needs the package installed (the Docker image has it; bash pthfa/install.sh in a venv) and the measured setup (docs/long-context.md). Otherwise the arm keeps its own default, fp8 on FlashInfer on mtp and the FA2 arm below on dflash2, and PTH_FA=0 restores it. Any other GPU, CTX=fast, CTX=huge and batch mode are unchanged. On the native RTX 3090 (TP=1, 250 W, vLLM 0.31.0, two boots per arm, #345), SPEC=mtp CTX=long decodes 84.7-84.8 tok/s at ~66k and 78.5 at ~100k against 75.5-75.6 and 68.8-68.9 with PTH_FA=0 (+12.2% and +14.0%), with the same cold TTFT at ~66k and a 1.9% smaller KV pool; the ~95-100 tok/s of row D above is the fp8 + FlashInfer arm. Only compute capability 8.6 has been run on a GPU (3090 and 3080 here), on vLLM 0.31.0, and the kernel is forward only.

This is the beginning of the README. Read the rest at the original. Read the full README ↗