Xinference (Xorbits Inference) is an open-source distributed inference platform.

Deploy and run models on heterogeneous hardware, from a local machine to a distributed cluster. Xinference includes 300+ built-in models for language, vision, embeddings, reranking, image, audio, video, and experimental world generation. Register your own models, including traditional ML models through the Flexible type. Manage deployments in the Web UI and use models through OpenAI-compatible APIs, the Python SDK, or the CLI.

Quick start · Key features · Comparison · Self-hosting · Documentation · Model catalog · Releases · Xinference Enterprise

Languages · English · 日本語 · 한국어 · Deutsch · Français · Español · Italiano · Português · 繁體中文 · 简体中文

Quick start

1. Start the server

Xinference downloads the model and installs any needed engine dependencies on the first launch.

pip install xinference
xinference-local --host 127.0.0.1

Open http://127.0.0.1:9997. On a fresh instance, use the setup token printed in the server log to create an administrator account, then sign in. First-run setup

2. Launch a chat model

In Launch Model, search for qwen3.5 and open its launch form. Click Recommend configuration to fill in settings for your hardware. Review the recommendation and click Deploy.

3. Make your first request

Open the deployed model in Running Models to try a prompt in the Web UI. The first instance uses qwen3.5 as its Model UID. If your instance shows a different UID, use that value for model below. Create a key under API Key Management to call the API.

export XINFERENCE_API_KEY="<your-api-key>"
curl --fail-with-body http://127.0.0.1:9997/v1/chat/completions \
  -H "Authorization: Bearer $XINFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5",
    "messages": [{"role": "user", "content": "Translate this sentence into Spanish: The model is ready."}],
    "enable_thinking": false,
    "max_tokens": 96
  }'

This is the beginning of the README. Read the rest at the original. Read the full README ↗