⭐ Help us reach more developers and grow the SIE community. Star this repo!
About
SIE is an open-source inference engine that runs the models behind every agent task through one API: search and retrieval, document-to-markdown conversion, structured output, content safety, the agent loop itself, and the translation, transcription, and vision calls around it. It replaces the patchwork of a separate model server per task with one system that serves 100+ models, loading each on demand.
- OpenAI-compatible API for drop-in migration:
/v1/embeddings, /v1/chat/completions, /v1/completions, /v1/responses
- Pre-configured model catalog: Stella, SPLADE, Qwen3, GLiNER, SigLIP, and more; embedding and retrieval models benchmarked on MTEB
- Serves multiple models simultaneously with on-demand loading and LRU eviction
- Ships Kubernetes and Helm deployment configs for the load-balancing gateway, KEDA autoscaling, and Grafana dashboards
- Integrates with LangChain, LlamaIndex, Haystack, DSPy, CrewAI, Chroma, Qdrant, Weaviate, and LanceDB
Tasks
One SIE cluster runs the inference behind a whole agent. Each task is a handful of swappable models; every name below links to its config, whose sie_id is the model name you pass to the SDK (hf_id is the Hugging Face repository the weights load from; the two usually match). Browse packages/sie_server/models/ for the full set.
| Task | What it does | Models |
|---|
| Search | Embed, match, and rerank to retrieve the right context. | bge-m3, splade-v3, colbertv2, qwen3-reranker |
| Document to markdown | PDFs, Office files, and scans become clean markdown. | lightonocr-3, lightonocr, glm-ocr, mineru, paddleocr-vl, docling |
| Structured output | Schema-valid JSON, extracted or generated. | , , , , , |
Quickstart
The quickstart uses the smallest model in each family so the first run is fast on a laptop. Swap in any model from the table above for production quality.
1. Start the server
# macOS (Apple Silicon) or Linux, native (requires Python 3.12)
pip install "sie-server[local]" && sie-server serve
# Linux, NVIDIA GPU
docker run --gpus all -p 8080:8080 \
-v sie-hf-cache:/app/.cache/huggingface \
ghcr.io/superlinked/sie-server:latest-cuda12-default
# Linux, NVIDIA GPU: Transformers 5 models (`:transformers` OCR profiles, GLiNER2.5-Decide, and TopK-Embed-V1)
docker run --gpus all -p 8080:8080 \
-v sie-hf-cache:/app/.cache/huggingface \
ghcr.io/superlinked/sie-server:latest-cuda12-transformers5
# Linux, NVIDIA GPU — SGLang vision OCR (default profiles of LightOnOCR-3, LightOnOCR-2, GLM-OCR, PaddleOCR-VL)
docker run --gpus all -p 8080:8080 \
-v sie-hf-cache:/app/.cache/huggingface \
ghcr.io/superlinked/sie-server:latest-cuda12-sglang-vision-extract
# Linux, CPU
docker run -p 8080:8080 \
-v sie-hf-cache:/app/.cache/huggingface \
ghcr.io/superlinked/sie-server:latest-cpu-default
Docker images are bundle-specific so dependency-incompatible model families stay isolated. Use the
sglang-vision-extract image for LightOnOCR-3, LightOnOCR-2, GLM-OCR, and PaddleOCR-VL, or the transformers5 image for
the :transformers profiles of LightOnOCR-3, LightOnOCR-2, and GLM-OCR, the GLiNER2.5-Decide models, and the TopK-Embed-V1 models; the
default image intentionally does not advertise them.
# in a second terminal
curl http://localhost:8080/readyz # expect: ok
The server speaks the OpenAI API out of the box, embeddings and generation alike (the cluster gateway serves /v1/chat/completions, /v1/completions, and /v1/responses). Your first call needs nothing but curl:
curl http://localhost:8080/v1/embeddings \
-H 'Content-Type: application/json' \
-d '{"model": "sentence-transformers/all-MiniLM-L6-v2", "input": "Hello world"}'
# {"object": "list", "data": [{"object": "embedding", "embedding": [-0.0344, 0.0310, ...
Each model's first call downloads its weights (progress appears in the server terminal). Later calls skip the
download; inference latency depends on the model, task, hardware, and batch size.
2. Install the SDK
pip install sie-sdk # Python
npm install @superlinked/sie-sdk # TypeScript (pnpm and yarn work too)
3. Generate embeddings, rerank, and extract entities
from sie_sdk import SIEClient
from sie_sdk.types import Item
client = SIEClient("http://localhost:8080")
# Generate embeddings
result = client.encode("sentence-transformers/all-MiniLM-L6-v2", Item(text="Hello world"))
print(result["dense"].shape) # (384,)
# Rerank search results
scores = client.score(
"cross-encoder/ms-marco-MiniLM-L-6-v2",
Item(text="What is machine learning?"),
[Item(text="ML learns from data."), Item(text="The weather is sunny.")],
)
print(scores["scores"][0]) # {'item_id': 'item-0', 'score': -7.1, 'rank': 0}
# Extract entities
result = client.extract(
"urchade/gliner_multi-v2.1",
Item(text="Tim Cook is the CEO of Apple."),
labels=["person", "organization"],
)
print(result["entities"][0])
# {'text': 'Tim Cook', 'label': 'person', 'score': 0.992, 'start': 0, 'end': 8, ...}
# Entities and a schema-valid record from one forward pass
result = client.extract(
"knowledgator/gliformer-base-v1",
Item(text="Refund request from Maria Lopez for order 4471 was denied: the item was returned after 45 days."),
labels=["person"],
output_schema={
"type": "object",
"properties": {
"order number": {"type": "string"},
"reason": {"type": "string"},
"decision": {"type": "string", "enum": ["approved", "denied", "escalated"]},
},
},
)
print(result["data"])
# e.g. {'order number': '4471', 'reason': 'the item was returned after 45 days', 'decision': 'denied'}
Typed decisions ask one or more typed questions about each item (a text, or a JSON object or conversation passed as
metadata={"state": ...}) and return an answer with probabilities per question in data. laya and
laya-typed-decisions calibrate the probabilities with their shipped temperatures; laya-multilingual ships none, so
its probabilities are the raw softmax. usage.input_tokens counts every (item, question) row the model encodes: the
state's tokens plus the question's, once per question. An item may use up to 32,768 row tokens (questions × max_len),
so the 1024-token laya-multilingual and laya-typed-decisions take up to 32 questions per request and laya up to 64.
result = client.extract(
"convaiinnovations/laya",
Item(text="Hi, we were billed twice for March. Please refund the duplicate today."),
output_schema={
"department": {"type": "choice", "instructions": "Which team should handle this?",
"criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages"}},
"refund_requested": {"type": "noul", "instructions": "Does the user ask for a refund?"},
"urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["low", "medium", "high"]},
},
)
print(result["data"]["department"]) # values are illustrative and rounded
# {'type': 'choice', 'choice': 'billing', 'probabilities': {'billing': 0.987, 'technical': 0.013}, 'confidence': 0.9}
The GLiNER2.5-Decide models (fastino/GLiNER2.5-Decide for English, GLiNER2.5-multi-Decide, and
GLiNER2.5-Decide-1B), served by the transformers5 image from step 1, take the same questions, GLiClass-style options={"label_groups": {...}}, or plain labels,
and return every option's probability. They read all of a call's questions next to the document in one row per item:
one forward pass answers them all, and each question's probabilities depend on the other questions sent with it. A
score question is read as the ordinal labels "0" to "k-1", each described by its criterion; a noul question as
"yes"/"no". usage.input_tokens counts the document tokens the model reads plus the questions' instructions and
criteria text; question ids and label names are not counted.
result = client.extract(
"fastino/GLiNER2.5-Decide",
Item(text="Guest in room 1408 says the AC has been out since yesterday and wants to move rooms tonight."),
output_schema={
"intent": {"type": "choice", "instructions": "What does the guest want?",
"criteria": {"room_change": "move to another room", "maintenance": "fix something", "checkout": None}},
"needs_human": {"type": "noul", "instructions": "Must a person act on this?"},
"urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["low", "normal", "high", "urgent"]},
},
)
print(result["data"]["intent"]) # values are illustrative and rounded
# {'type': 'choice', 'choice': 'room_change',
# 'probabilities': {'room_change': 0.871, 'maintenance': 0.085, 'checkout': 0.045}, 'confidence': 0.57}
Text generation runs on the GPU generation image; stop the first server, then start this one on the same port:
# Linux, NVIDIA GPU (for generation on Apple Silicon via MLX, see the docs below)
docker run --gpus all -p 8080:8080 \
-v sie-hf-cache:/app/.cache/huggingface \
ghcr.io/superlinked/sie-server:latest-cuda12-sglang
result = client.generate(
"Qwen/Qwen3-0.6B",
"Reply with a single word: the capital of France.",
max_new_tokens=16,
temperature=0.0,
)
print(result["text"]) # Paris
For generation on Apple Silicon (MLX), the TypeScript walkthrough, and every configuration in between, see the quickstart guide, TypeScript SDK docs, and SDK reference.
Production
The same code works against a production cluster. SIE ships a load-balancing gateway and Kubernetes deployment surface, including Helm charts, KEDA autoscaling (scale to zero), and Grafana dashboards. Public Terraform modules are maintained separately for Alibaba Cloud ACK, EKS, AKS, and GKE. Not just the server, the whole stack. All Apache 2.0.
# pick one values overlay: values-ack.yaml / values-aws.yaml / values-aks.yaml / values-gke.yaml
# (pin a chart version for reproducible installs, e.g. --version 0.7.3)
helm upgrade --install sie-cluster oci://ghcr.io/superlinked/charts/sie-cluster \
--namespace sie --create-namespace \
--set hfToken.create=true \
--set hfToken.value=YOUR_HF_TOKEN \
-f https://raw.githubusercontent.com/superlinked/sie/main/deploy/helm/sie-cluster/values-gke.yaml
See the deployment guide.
Telemetry
sie-server sends anonymous usage telemetry so the maintainers can see which versions, platforms, and GPUs are in use. It is on by default in releases before 0.13.0. From 0.13.0 it is off unless SIE_TELEMETRY_ENABLED=1 is set. Until then, a server that sends it only because of the default logs a notice at startup.
Each event is a JSON POST to https://telemetry.superlinked.com/api/telemetry (SIE_TELEMETRY_URL overrides the endpoint) with these fields:
| Field | Value |
|---|
worker_id | A random UUID generated on first start and stored in the server data directory (SIE_DATA_DIR) |
sent_at | The send time (UTC) |
event | init, update, or terminate |
sie_version | The installed sie-server version |
variant | SIE_VARIANT: the image platform and bundle, set by the Helm chart (null otherwise) |
os, arch | Operating system and CPU architecture |
gpus | GPU model names |
deployment_env | SIE_DEPLOYMENT_ENV (default unknown) |
The receiver derives country, region, and city from the request's source IP address and does not store the address. No request data, hostnames, or file paths are sent.
init is sent at start, the first update 45 to 75 minutes later, then one update every 55 to 65 minutes, and terminate at shutdown.
- Opt in:
SIE_TELEMETRY_ENABLED=1.
- Opt out:
SIE_TELEMETRY_DISABLED=1 or DO_NOT_TRACK=1, which always win, or SIE_TELEMETRY_ENABLED=0.
SIE_TELEMETRY_ENABLED and SIE_TELEMETRY_DISABLED accept 1, true, yes, on and 0, false, no, off, in any case. Any other value of either turns telemetry off with a warning.
Every start logs the state and where it came from, for example telemetry: on (default) or telemetry: off (DO_NOT_TRACK). The Helm chart's telemetry.enabled default also changes to false in 0.13.0; see the chart README. Policy: superlinked.com/policies/telemetry.
Explore
Model catalog: every model is a config in packages/sie_server/models/; pass its sie_id to the SDK.
Integrations: setup guides for all nine framework and vector-store integrations, in Python and TypeScript.
Examples: An end-to-end project gallery.
MCP edge: offload document, image, and structured-output work from Claude and other MCP clients to your cluster and save agent tokens.
Why we built SIE: The motivation, told at AI Engineer Europe 2026.
Development
Read CONTRIBUTING.md for the complete development
workflow and AGENTS.md for repository automation boundaries.
Install mise, then bootstrap the
versioned Python, Rust, Node.js, and Helm toolchains from the repository root:
./tools/init.sh
The common development checks and local server are available as mise tasks
(mise tasks lists them all):
mise run test
mise run lint
mise run typecheck
mise run serve
mise run rust-check
mise run rust-test
mise run gateway-test
mise run server-sidecar-test
mise run helm -- dependencies
mise run helm -- lint --set payloadStore.enabled=false
mise run helm -- template --set payloadStore.enabled=false
The Python workspace uses the committed root lock. Package membership is
explicit in pyproject.toml; a package joins the workspace only in the same
change that adds its complete source. Native audio is always an opt-in build:
install cmake, then run
mise exec -- uv sync --frozen --project . --all-packages --all-extras.
