qwen38-flash

Note: The bulk of this repository (program, build and benchmark scripts, research, documentation and website) was created by Claude Fable 5.1 (Anthropic), directed and reviewed by the author. The measurements come from real runs on the hardware described.

A terminal program with which you set up, start, monitor and measure Qwen3.8-Flash-Next on an AMD Strix Halo machine (Ryzen AI MAX+ 395 / Radeon 8060S, 128 GB unified memory) with llama.cpp. It takes the decisions off your hands that are difficult with this model: Which quant fits into memory? Which startup parameters make sense? Does speculative decoding (MTP) run? How many users at the same time?

Everything is measured on such a machine. The results and the reasoning are in docs/RESEARCH.md.

What the program can do

• Configure: model quant, context length, KV cache, batch sizes, MTP draft head, thinking mode, sampling, network. On the right you immediately see the estimated memory requirement and the finished command line.
• Protect memory: the start is refused if the configuration does not fit into memory. A guard stops the server before the kernel kills processes because of memory shortage.
• Start and observe: server log, load time, buffer sizes, tokens per second, acceptance rate of the draft head, test prompt.
• Measure: llama-bench, server measurement with MTP, automatic search for the best MTP settings and a multi-user test with up to 8 simultaneous requests.
• Export: start script and systemd unit for operation without the program.
• Presets: ready-made configurations for maximum quality, speed, long context, chat without thinking, a coding agent, and eh-team for several agents at once (7 slots of 262k, one each, with the prompt cache set up so a follow-up turn hits it to 99.5 %).

The program changes nothing in your model files and nothing outside its own directory.

Requirements

What	Why
AMD Strix Halo (gfx1151) with 128 GB, Linux	everything is measured for that; other machines with a lot of unified memory should work, but are not tested
ROCm/HIP 7.x with clang++, cmake, ninja, git	to build the two llama.cpp variants
Kernel parameters amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856	otherwise the GPU may use only part of the RAM
Python 3.12 or newer, uv	for the terminal program
hf (Hugging Face CLI)	to download the models
about 200 GB free space on a fast NVMe	models 73 to 104 GB per quant

Installation

git clone <URL of this repo> qwen38-flash
cd qwen38-flash
uv sync                      # creates .venv and installs Textual & co.

Download models

# Best quant that fits into 128 GB with MTP (engine EngramHalo):
hf download unsloth/Qwen3.8-Flash-Next-GGUF --include 'UD-Q4_K_XL/*'
# Smaller alternatives:
hf download unsloth/Qwen3.8-Flash-Next-GGUF --include 'UD-IQ4_XS/*'
hf download unsloth/Qwen3.8-Flash-Next-GGUF --include 'UD-IQ3_XXS/*'
# MTP draft head (only this one matches the builds here):
hf download dzannotti/Qwen3.8-Flash-Next-MTP-GGUF Qwen3.8-Flash-Next-MTP-Q4_K_M.gguf

The program finds the files in the Hugging Face cache by itself. If they are elsewhere, set QWEN38_MODEL_DIRS=/path/a:/path/b and QWEN38_MTP_DIRS=/path/mtp.

Build engines

engine/fetch.sh                 # fetches llama.cpp and EngramHalo.cpp from public repos and patches them
engine/build-engramhalo.sh      # recommended engine (Strix Halo fork)
engine/build.sh hip             # second engine: llama.cpp + MTP patch

Both builds end up under engine/build-*/bin/. You register your own llama.cpp build with QWEN38_ENGINES=name=/path/to/llama.

Usage

./run.sh                        # terminal program
./run.sh presets                # all presets
./run.sh show --preset eh-qualitaet    # show command line and memory balance
./run.sh run  --preset eh-qualitaet    # start server without the program
./run.sh bench-parallel --users 8      # multi-user benchmark without the program
./run.sh export --preset eh-team --host 127.0.0.1 --port 18080 \
    --api-key-file state/api-keys.txt --new-keys 5 --install    # service behind the router

Running as a service on the network

export writes a start script and a systemd user unit to scripts/, and with --install also to ~/.config/systemd/user/. The unit starts the server under bench/memguard.py, because GPU memory (GTT) shows up in no cgroup limit — MemoryMax would not protect anything here, only MemAvailable does. A guard kill is not retried endlessly: three attempts in ten minutes, then the service stays down.

systemctl --user daemon-reload
systemctl --user enable --now qwen38-eh-team.service
systemctl --user status qwen38-eh-team.service
journalctl --user -u qwen38-eh-team.service -f

A user unit runs only while the user is logged in. For a start at boot: sudo loginctl enable-linger $USER.

--new-keys N writes N random keys to the file given by --api-key-file, one per line, with mode 0600; existing files are never overwritten. The server takes them with --api-key-file, so the keys do not appear in the process list. Clients send Authorization: Bearer <key>; without one, or with a wrong one, the server answers 401. Add or revoke a key by editing the file and restarting the service.

halogen-flash-server and the router

halogen/ sets up halogen-flash-server — its own HIP engine rather than a llama.cpp fork — as a second service. Both servers hold most of the machine, so they run in turn, and router/ puts one endpoint in front of them on port 8080: same API keys, same model id qwen3.8-flash, same defaults for whatever a request leaves open, and it waits during a switch instead of failing. The client router (LiteLLM) needs one entry and never learns which server answered.

router/umschalten.sh halogen     # or: llama, status

Both give 262,144 tokens per request. halogen prefills about four to five times faster, scales to about 69 t/s with four users and keeps its prompt cache on disk across restarts; details in halogen/README.md.

Vision

The model is text-only on its own — the weights contain no image encoder. Vision comes from a separate projector file that unsloth ships in the same repo:

hf download unsloth/Qwen3.8-Flash-Next-GGUF mmproj-F16.gguf

Anything named mmproj*.gguf next to the weights (or in state/mmproj/, or in QWEN38_MMPROJ_DIRS) is picked up automatically; mmproj: "auto" in a preset then adds --mmproj to the command line, and /props reports vision: true. The projector is qwen3vl_merger and costs about 1.1 GB. Verified: a test image with shapes and the text "STRIX HALO 47" was described correctly, colours and text included.

In LiteLLM the server side is only half of it — for an OpenAI-compatible endpoint LiteLLM does not assume image support and rejects image content before it ever reaches the server. Declare it:

  - model_name: qwen38-flash
    litellm_params:
      model: openai/qwen3.8-flash
      api_base: http://<lan-ip>:8080/v1
      api_key: os.environ/QWEN38_KEY
    model_info:
      supports_vision: true

--host 0.0.0.0 opens the server to the whole network. The keys are the only protection, and plain HTTP sends them unencrypted — so use it on a trusted network, or put TLS in front (--ssl-key-file / --ssl-cert-file).

In the program: choose a preset and press "Apply preset". On the right are the memory balance and the command. F5 starts the server, F6 stops it, F9 exports a start script, Ctrl+S saves the configuration. Tabs: Configuration, Server, Benchmark, System, Help.

The server is then reachable at http://<host>:8080 with the OpenAI-compatible API (/v1/chat/completions), including the web interface of llama.cpp.

The most important findings

Memory. Every quant contains the same 26.8 GB table for n-gram embeddings. Normal llama.cpp loads it completely into RAM on ROCm, in addition to the weights in the GPU. That is why the best quant (UD-Q4_K_XL) with MTP does not fit into 128 GB there. The fork EngramHalo.cpp leaves the table on the NVMe (only about 3 GB in RAM) and still loads in under 30 seconds. With that, UD-Q4_K_XL runs with MTP at about 85 GB memory requirement.

Speed (one user, 32k context, KV cache q8_0):

Engine	Quant	MTP	Decode	Memory
EngramHalo	UD-Q4_K_XL	on	33–41 t/s	85 GB
EngramHalo	UD-IQ4_XS	on	36 t/s	69 GB
EngramHalo	UD-IQ3_XXS	off / on	23 / 34 t/s	53 / 57 GB
llama.cpp + patch	UD-IQ4_XS	on	34 t/s	93 GB
llama.cpp + patch	UD-Q4_K_XL	on	does not fit	over 107 GB

Several users (UD-IQ4_XS, without MTP): 1/2/4/8 simultaneous requests give 20/32/42/50 tokens per second in total. MTP only pays off with one user; with 8 users it is only 35 instead of 50 t/s with MTP.

Further decisions: ROCm instead of Vulkan (Vulkan collapses with MTP on this model). Only the dzannotti MTP head matches the builds; the unsloth head needs the unsloth fork. The environment variable LLAMA_ATTN_ROT_DISABLE is not needed. reasoning_effort only knows xhigh, medium and low.

Directories

Path	Content
qwen38tui/	the program (configuration, memory estimation, model detection, GGUF reader, server control, benchmarks, interface)
engine/	fetch.sh, build scripts, patches; after the build build-engramhalo/ and build-hip/
bench/	measurement scripts with memory guard, context_limits.py (how many simultaneous contexts fit per quant) and the raw results of this machine
bench/quality/	agent benchmark Terminal-Bench-Mini-20: fetch script, runner, analysis (see bench/quality/README.md)
docs/	RESEARCH.md (research with sources), QUALITY-BENCHMARKS.md (which quality benchmarks are possible in 8 hours), TERMINAL-BENCH.md (results of the agent benchmark), HELP.md (help inside the program) and the documentation website (HTML, published via GitHub Pages from this directory)
tests/	uv run pytest -q
state/	own profiles, logs, measurement results (created on first start)

Documentation website

The directory docs/ contains a static website with all decisions, measurements and sources (index.html, decisions.html, memory.html, measurements.html, terminal-bench.html, guide.html, research.html). Publishing: in the repository settings under "Pages" choose the branch main with the directory /docs. Repository: https://github.com/kksoftwareag/strix-halo-qwen-3.8-flash-next. No build step is needed; .nojekyll makes sure that GitHub delivers the files unchanged.

Known limits

• Only tested with ROCm/HIP on gfx1151. A Vulkan build is prepared, but not measured.
• MTP with several slots is not good on this architecture; the program switches it off in the multi-user test.
• The memory requirement is estimated. The estimate is calibrated to the measurements, but can deviate with unusual settings. The guard catches that.
• MTP support for this model is not yet included in llama.cpp. The patches here follow the open pull requests and have to be adapted for new llama.cpp versions.

Thanks

• llama.cpp and the authors of the qwen4exp support
• EngramHalo.cpp for the Strix Halo fork
• dzannotti for the MTP draft head and the patch
• unsloth for the quantizations
• Qwen for the model
• Textual for the terminal interface

License

This project is under the European Union Public Licence v. 1.2 (EUPL-1.2), see LICENSE. The models, llama.cpp, EngramHalo.cpp and the third-party patches are under their own licenses.