Closira: ML-Driven IT Helpdesk Assistant
An IT helpdesk assistant that decides when an employee needs a human. A multi-task DistilBERT reads each message for intent and emotion, and a calibrated risk router decides whether to keep helping or hand off with a ticket.
Live demo | Model on Hugging Face | Detailed results
Results
| |
|---|
| Intent accuracy | 97.7% (7 classes, baseline 34.1%) |
| Emotion accuracy | 80% (6 classes, baseline 39.6%) |
| Escalation | 80% recall at 7.8% false alarms (TF-IDF: 76% at 13.0%) |
| Served model | ONNX int8, 66.8 MB, 11 ms per message |
| Tests | 125 automated tests |
How it works
message -> DistilBERT (intent + emotion) -> risk router -> keep helping or hand off
- The model scores, the router decides. Routing is plain Python rules in
router.py; the LLM (Groq) only extracts details and writes replies.
- Slot filling: asks only for missing details (issue, error, device, impact), one at a time.
- SOP-grounded fixes: self-help steps come only from
sop.json; a ticket is created if they don't work.
- Risk-based priority: tickets are Critical, High, Medium or Low based on risk and impact.
Datasets
| Dataset | Used for | Rows used |
|---|
| Banking77, CLINC150, SNIPS | Intent head | mapped to 7 support intents |
| Twitter Customer Support (TWCS) | Intent head (complaint, feedback, appreciation) | keyword-labelled tweets |
| GoEmotions | Emotion head | single-label comments, mapped to 6 emotions |
| 616 IT helpdesk messages | Escalation test only (never trained on) | 308 held out |
In total: 39,861 intent and 31,979 emotion training rows. Original labels were mapped to Closira's labels in intent_map.json and emotion_map.json.
Intents: information_request, issue_report, action_request, complaint, feedback, appreciation, chit_chat
Emotions: angry, frustrated, confused, sad, happy, neutral
Model
distilbert-base-uncased with one shared encoder and two heads (intent and emotion) on the [CLS] token.- Trained with alternating batches (intent, then emotion), class-weighted cross-entropy, AdamW (lr 2e-5), early stopping. Best epoch 3 of 5, about 11 minutes on a Colab T4.
- One encoder instead of two models: about 66M parameters instead of 133M, and one forward pass per message.
- Exported to int8 ONNX (266 MB to 66.8 MB, 112 ms to 11 ms) with the same escalation results.
Evaluation
- Compared against keyword rules and TF-IDF + logistic regression on the same splits.
- Escalation thresholds were tuned on one half of the 616 helpdesk messages and reported on the other half.
- Limitations: the 616 messages are LLM-generated, so real-world recall may be lower; most of the recall gain came from keyword rules, while the model mainly reduced false alarms. Next: a conversation-level model for repeated confusion across turns.
Run locally
pip install -r backend/requirements.txt
uvicorn backend.main:app --reload --port 8000
cd frontend && npm install && npm run dev
Needs GROQ_API_KEY and DATABASE_URL. Training and evaluation commands are in backend/ml/results/README.md.
