⭐ Help us reach more developers and grow the SIE community. Star this repo!
About
SIE is an open-source inference engine that runs the models behind every agent task through one API: search and retrieval, document-to-markdown conversion, structured output, content safety, the agent loop itself, and the translation, transcription, and vision calls around it. It replaces the patchwork of a separate model server per task with one system that serves 100+ models, loading each on demand.
- OpenAI-compatible API for drop-in migration:
/v1/embeddings, /v1/chat/completions, /v1/completions, /v1/responses
- Pre-configured model catalog: Stella, SPLADE, Qwen3, GLiNER, SigLIP, and more; embedding and retrieval models benchmarked on MTEB
- Serves multiple models simultaneously with on-demand loading and LRU eviction
- Ships Kubernetes and Helm deployment configs for the load-balancing gateway, KEDA autoscaling, and Grafana dashboards
- Integrates with LangChain, LlamaIndex, Haystack, DSPy, CrewAI, Chroma, Qdrant, Weaviate, and LanceDB
Tasks
One SIE cluster runs the inference behind a whole agent. Each task is a handful of swappable models; every name below links to its config, whose sie_id is the model name you pass to the SDK (hf_id is the Hugging Face repository the weights load from; the two usually match). Browse packages/sie_server/models/ for the full set.
| Task | What it does | Models |
|---|
| Search | Embed, match, and rerank to retrieve the right context. | bge-m3, splade-v3, colbertv2, qwen3-reranker |
| Document to markdown | PDFs, Office files, and scans become clean markdown. | lightonocr-3, lightonocr, glm-ocr, mineru, paddleocr-vl, docling |
| Structured output | Schema-valid JSON, extracted or generated. | gliner2, gliner-relex, gliformer, , , |
Quickstart
The quickstart uses the smallest model in each family so the first run is fast on a laptop. Swap in any model from the table above for production quality.
1. Start the server
This is the beginning of the README. Read the rest at the original. Read the full README ↗
