A GLiNER bi-encoder inference service in Rust: zero-shot named entities over HTTP, CPU-first, on ONNX Runtime,
with the same predictions as the PyTorch reference.
An EDamLabs binding: a port of a third-party model that keeps its upstream's predictions and follows its releases, shipped as a service (EDamLabs families).
Labels are free text per request; the label tower encodes each once and caches it, the text tower scores every span
of up to 12 words against them. Offsets are characters, not bytes. Served: knowledgator/gliner-bi-base-v2.0
(ModernBERT text encoder, BGE-small label encoder), fp32 — the only precision that keeps PyTorch's predictions.
text ─▶ words ─▶ windows (max_len, overlap) ─▶ text.onnx ◀─ label embeddings ◀─ labels.onnx ◀─ LRU cache
│
spans > threshold ─▶ greedy, half-open, non-overlapping ─▶ entities
| |
|---|
| parity | 73/73 entities identical to PyTorch GLiNER 0.2.29 on the public fixture (Jaccard 1.0, max score drift 6.6e-7) |
| latency | 22.7 ms per ~100-character text, 108 ms per ~570-character text (8 threads, i9-14900K P-cores) |
| memory | 1.0 GB peak RSS; image 94 MB (distroless, model mounted) |
Quickstart
make install # pinned Rust, uv, prek, cargo-llvm-cov, cargo-about, hooks
make export MODEL=knowledgator/gliner-bi-base-v2.0 # PyTorch → data/models/gliner-bi-base-v2.0/1 (ONNX artifact), served
make serve # :5800
curl -s localhost:5800/ner -H 'content-type: application/json' \
-d '{"text": "José Saramago vivió en Lanzarote.", "labels": ["person", "location"]}'
{"entities":[{"start":0,"end":13,"text":"José Saramago","label":"person","score":0.70},
{"start":23,"end":32,"text":"Lanzarote","label":"location","score":0.74}]}
Docker: make image && make up (the model registry is mounted read-only at /models; nothing is downloaded at run
time).
Training
The same binary in a second image, gliner-train (make image-train, PyTorch CPU or TORCH=cu128), with
train.enabled: upload datasets and private eval sets as NDJSON, POST /train, and the job does the rest — leakage
check, fine-tuning, export, parity with PyTorch (≥ 99 % identical predictions), regression against the served model on
every eval set with paired-bootstrap intervals, a gate, registration as a new version, and a hot swap if asked and
earned. Without the role, those routes answer 501. Flow, registry and gate: docs/train.md.
curl -s -X PUT localhost:5800/evals/support --data-binary @support.eval.jsonl
curl -s -X PUT localhost:5800/datasets/support --data-binary @support.train.jsonl
curl -s localhost:5800/train -H 'content-type: application/json' -d '{"dataset": "support", "auto_promote": true}'
curl -s localhost:5800/train/1 # state, step, log tail, report, version
API
| Route | |
|---|
POST /ner | {text, labels[], threshold?} → {entities: [{start, end, text, label, score}]} |
POST /ner/batch | NDJSON in, NDJSON out, streamed: {id, text, labels?, threshold?} per line → {id, entities} or {id, error} as each completes |
GET /health | liveness and version; answers while the model loads |
GET /ready | 200 once the model is loaded, 503 before |
GET /models | {model, version, labels_cached, served}; version hashes the model files, served is the registry version |
PUT /datasets/{name}, PUT /evals/{name} | NDJSON {text, entities: [{start, end, label}]}, checked line by line, stored atomically → {name, lines, sha256} (train role) |
GET /datasets, GET /evals | the stored sets (train role) |
POST /eval | {set, version?} → strict, lenient and per-label P/R/F1 (train role) |
POST /train, GET /train[/{id}], DELETE /train/{id} | a train job: start (202), follow, cancel (train role) |
POST /promote | {version}: hot swap to a registered version; the champion keeps serving if it fails to load (train role) |
Details, errors and the batch protocol: docs/api.md. Configuration: docs/configuration.md.
The artifact contract and the export: docs/model.md. Training: docs/train.md. Development, gates and workflows:
docs/development.md.
License
MIT. ONNX Runtime (MIT) is linked statically; every Rust dependency is listed in
THIRD_PARTY_LICENSES.md. Model weights keep their own licence (knowledgator/gliner-bi-base-v2.0: Apache-2.0).
