OrchestrANT

Docs site

OrchestrANT is the family's Python AI package and its LLM benchmark lab. The orchestrant package orchestrates AI workloads: camera pipelines, detection with yolo-monitor, streaming, and system and GPU metrics. Beside it live the lab (benchmarks/ and the orchestrant-bench runner), the Flet viewer for its results, and the template a new Python AI project copies its uv/ruff/ty setup, Cython wheel build, Sphinx docs and CI wiring from. The four stay in one repository by owner decision (2026-09-15).

Dependency updates: run bash scripts/linux/renovate-local.sh to report, --apply --dry-run to plan, and --apply to update. A real apply initializes a missing pinned hub and follows each direct submodule's remote default branch. CI checks the wrapper and config; see the shared Renovate guide. Torch/vision updates require the accepted hub pair to be pinned and committed first.

Highlights

• Installable packages with their own Python: MSI, MSIX and zip for Windows x64/arm64; .deb, AppImage and tarball for Linux x86_64/arm64, with the model's locked licence and notice, and a third-party notice for every bundled component, checked in each package.
• yolo-monitor runs ModelANT's DEIM-D-FINE-N (Apache-2.0) on CUDA, MIGraphX, DirectML or CPU when provided by ONNX Runtime; --self-test needs no camera.
• orchestrant-bench measures throughput, time-to-first-token, decode rate and correctness of any OpenAI-compatible endpoint.
• GPU monitoring for two vendors through one GPUProbe: NVML for NVIDIA, ADL or amdgpu sysfs for AMD.
• Windows DirectML candidate: run the on-request Windows x64 DirectML candidate workflow for packages built with the registry's NVIDIA and AMD images. The optional GPU profiles need self-hosted Windows runners labelled directml-gpu-nvidia and directml-gpu-amd; a candidate built without them has not been hardware-tested.
• The shipped wheels compile every module with Cython; the free-threaded cp314t wheel runs them with the GIL off.
• Static analysis is a gate: ruff, ty, bandit, vulture and codespell fail CI on any finding.

GPU package candidates

The on-demand Linux GPU inventory and Windows DirectML workflows target NVIDIA x64/arm64 and AMD x64 on Linux, and NVIDIA/AMD x64 on Windows. Open a successful run and download the matching vendor and architecture from its Artifacts section (sign-in required). Linux outputs are inventory candidates, not release packages; Windows outputs are also candidates separate from releases. They need matching host GPU drivers and hardware verification. The recorded infrastructure pin currently blocks both workflows; see getting started for formats and remaining checks. CPU packages remain available through the existing lanes. Both candidate workflows can also be called by another workflow; GPU profiling stays opt-in and inventory-only outputs remain separate from candidates. GPU uploads use run-attempt-specific names to avoid replacing an earlier candidate. Select one Linux gpu-tuple or Windows gpu-vendor; overlapping reusable calls need distinct invocation-scope values.

Quick start

git clone --recurse-submodules --branch develop https://github.com/Kataglyphis/OrchestrANT.git
cd OrchestrANT
python third_party/ANTfrastructure/linux/scripts/lib/fetch-data-lock.py --lock resources/models/models.lock   # once: the model
uv run --extra app yolo-monitor --self-test    # one inference on a blank frame, a JSON report

The packages, the library install, a camera, the tests and the development setup: docs/source/getting-started.md.

Documentation

Topic	Read
Getting started: packages, the library, demos, tests, development	docs/source/getting-started.md
The package, yolo-monitor, the model and the extras	docs/source/overview.md
Language models on the chain GenAI: orchestrant-chat, orchestrant-serve, orchestrant-text, orchestrant-docs, yolo-monitor --describe	docs/source/language-models.md
orchestrant-bench, the LLM endpoint runner	docs/source/benchmark.rst
System and GPU monitoring	docs/source/monitoring.md
The LLM benchmark lab: tools, results, the gateway	benchmarks/README.md
The lab's roadmap	benchmarks/docs/llm-benchmark-roadmap.md
Connecting a model to the NAS	benchmarks/docs/nas-document-ai.md
The model weights and their lock	resources/models/README.md

Contributing and contact

OrchestrANT is one of the Kataglyphis repositories, which share one way to contribute and one contact. Licence: MIT, see LICENSE.