hipfire

What hipfire is

hipfire is an inference engine for AMD GPUs. It ships its own Rust runtime, model implementations, quantization formats (MQ3–MQ6, MQ4R, MFP4), typed kernel dispatch and HIP kernels. ROCm is loaded dynamically. There is no Python, PyTorch, CUDA translation layer or third-party inference engine in the hot path.

You get a curated model registry, a CLI (pull, run, chat, serve, bench, quantize) and an OpenAI-compatible HTTP daemon. Speculative decoding (DFlash draft models, native MTP heads), multi-GPU expert and tensor parallelism and the Redline replay path ship with the engine.

Quickstart

hipfire pull qwen3.8:27b-mq4-xts
hipfire serve qwen3.8:27b-mq4-xts -d
hipfire chat qwen3.8:27b-mq4-xts

This downloads about 15 GB and needs a 24 GB+ card or a Strix Halo system. On smaller cards, use qwen3.5:4b or qwen3.5:9b with the same commands. Pick a model for your GPU lists recommendations for each GPU tier.

For one-shot inference:

hipfire run qwen3.8:27b-mq4-xts "What is the capital of France?"

Qwen3.8-Flash (qwen3.8:flash-next) is a MoE model. The tag resolves to the split base qwen3.8-flash-next-base.mq4-xts (about 70.9 GB), plus a shared per-token-embedding (PLE) sidecar (29.1 GB for q4 rows, 54.4 GB for q8 rows) and, on gfx1201, a 213 KB expert hot-set file. The old 125 GB monolith (qwen3.8-flash-next.mq4-xts, formerly qwen3.8-flash-next-gptq3.mq4) stays published for existing clients:

hipfire pull qwen3.8:flash-next
hipfire serve qwen3.8:flash-next -d

A 32 GB card alone is not enough for Qwen3.8-Flash. It needs a 128 GB Strix Halo system, or a discrete card with enough host RAM for the mapped experts (the 0.4.4 release smoke ran it on an R9700 and on a 7900 XTX), or two R9700s (EP2). Native context is 262K tokens, subject to available memory.

Image generation (RDNA3/3.5 measured):

hipfire pull flux.schnell:1
hipfire img --out x.png flux.schnell:1 "a red cube on a wooden table"

The daemon serves an OpenAI-compatible API on 127.0.0.1:11435. It listens on loopback only unless you set serve.host.

curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.8:27b-mq4-xts",
       "messages": [{"role": "user", "content": "What is 2+2?"}],
       "max_tokens": 256,
       "chat_template_kwargs": {"enable_thinking": false}}'

Qwen models think by default, and max_tokens counts the <think> span as well as the answer. If thinking is on and the cap is small, content can come back empty with finish_reason: "length". Turn thinking off as shown above, or raise max_tokens. Qwen3.8 and Ornith 1.5 default to medium reasoning effort. See docs/SERVE.md.

What's new in 0.4.4

Every figure in this section comes from the v0.4.4 release notes and the CHANGELOG 0.4.4 entry. Each is a measurement on one named fixture and card, not a general throughput claim. Rows marked contributor-measured were not re-run by the maintainers on the contributor's card; the merge gates ran the build, the CPU tests and a smoke on a different card. The CHANGELOG entry has the fixtures, methods, kill switches and upgrade notes.

以上为 README 的开头部分。其余内容请阅读原文。 查看完整 README ↗