OpenVINO™ Model Server
High-performance model serving for Generative AI and classic deep learning — powered by OpenVINO and optimized for Intel hardware.
What is OVMS?
OpenVINO Model Server (OVMS) is a production-grade, C++ inference server that exposes ML models over standard network APIs. It serves both Generative AI, Agentic workloads (LLMs, VLMs, image generation, audio) and classic deep learning models (object detection, classification, OCR, and more).
- OpenAI-compatible API for text generation, embeddings, image generation, and audio
- KServe APIs for classic model inference
- Runs anywhere — Docker, bare metal, Kubernetes/OpenShift, Windows
- Intel-optimized — CPU, GPU, NPU acceleration via OpenVINO
Quick Start
Serve an LLM with OpenAI-compatible API
On Linux (Docker):
mkdir -p ${HOME}/models
# Model is downloaded automatically from HuggingFace to models folder
docker run --rm -p 8000:8000 \
--user $(id -u):$(id -g) -v ${HOME}/models:/models:rw \
openvino/model_server:latest \
--source_model OpenVINO/Qwen3-4B-int4-ov \
--model_repository_path /models \
--rest_port 8000
For GPU acceleration, use the latest-gpu image tag and pass --device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1) to expose the Intel GPU device.
On Windows (binary package):
mkdir c:\models
ovms.exe --source_model OpenVINO/Qwen3-4B-int4-ov --model_repository_path c:\models --rest_port 8000
Query the model:
pip install openai
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
stream = client.chat.completions.create(
model="OpenVINO/Qwen3-4B-int4-ov",
messages=[{"role": "user", "content": "What are the 3 main tourist attractions in Paris?"}],
stream=True,
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
LLM QuickStart
Serve a Classic Model with KServe API
Download the model:
curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.bin -O
curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.xml -O
On Linux (Docker):
docker run --rm -d -u $(id -u) -v ${PWD}:/models -p 9000:9000 \
openvino/model_server:latest \
--model_name resnet --model_path /models/resnet50.xml \
--mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" \
--port 9000
For GPU acceleration, use the latest-gpu image tag and pass --device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1) to expose the Intel GPU device.
Windows (binary package):
ovms --model_name resnet --model_path resnet50.xml --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" --port 9000
Run inference with a sample client
pip install numpy tritonclient[grpc]
curl -L -o image.jpeg https://github.com/openvinotoolkit/model_server/blob/main/demos/common/static/images/bee.jpeg?raw=true
import numpy as np
import tritonclient.grpc as grpcclient
with open("image.jpeg", "rb") as f:
image_bytes = f.read()
client = grpcclient.InferenceServerClient(url="localhost:9000")
inputs = [grpcclient.InferInput("image", [1], "BYTES")]
inputs[0].set_data_from_numpy(np.array([image_bytes], dtype=object))
outputs = [grpcclient.InferRequestedOutput("output")]
result = client.infer(model_name="resnet", inputs=inputs, outputs=outputs)
output = result.as_numpy("output") # (1, 1000) FP32
print("Top-1 class index:", int(np.argmax(output[0])))
Vision model QuickStart
Features
Generative AI
This is the beginning of the README. Read the rest at the original. Read the full README ↗
