EU Rail Regulation Assistant
A question-answering assistant over EU railway interoperability law: the Interoperability Directive and six Technical Specifications for Interoperability (TSIs). Every answer cites the exact passage it comes from, and the assistant refuses when its sources don't cover the question. An evaluation harness runs on GitHub Actions and fails the build when answer quality, grounding or refusal behaviour drops below fixed thresholds.
Live demo: https://rail-rag.calmcliff-0073b8b6.switzerlandnorth.azurecontainerapps.io
Demo video (90 s): https://youtu.be/CVuVkNnhasc
Used by: rail-compliance-agent, a tool-using agent that runs this assistant's retrieval as one of its tools to check a train design spec against the PRM TSI, and is evaluated on its tool use (it pins this repo at one commit).
The demo scales to zero when idle, so the first request after a pause is slower: 6.7 s, measured once after 12 minutes idle (1.0 s once warm).
Stack: Python 3.12 · LangGraph · FastAPI · Azure AI Search hybrid search, BM25 + vectors (with a local Qdrant fallback) · ONNX bge-small-en-v1.5 embeddings · Gemini (generator) · gpt-oss-120b via Groq (judge) · RAGAS · Langfuse · Docker · Azure Container Apps · GitHub Actions. Runs on free tiers plus a few cents of Azure for Students credit: €0.18 in the first 3 days, mostly setup traffic. No paid plan or card is involved.
Results
Measured on the 20-question golden set (eval/golden_set.jsonl), from GitHub Actions run 38838060201 at commit 097bfba, the commit the live demo runs. This is the gated run for dense-only retrieval; hybrid retrieval is measured below. The raw numbers are in eval/results/latest.json.
| Metric | How it's measured | Value | Gate |
|---|
| Faithfulness | RAGAS; judge gpt-oss-120b (a different model family from the generator) | 1.00 | ≥ 0.80 |
| Answer relevancy | RAGAS | 0.90 | ≥ 0.70 |
| Context precision | rank-weighted, against gold passage ids | 0.94 | ≥ 0.50 |
| Context recall | against gold passage ids | 0.89 | ≥ 0.70 |
| Grounding | answers cite only the documents the question is about | 1.00 | ≥ 0.90 |
| Answerable questions answered | 16 answerable questions | 0.94 (15/16) | ≥ 0.80 |
| Unanswerable questions refused | 4 out-of-scope / near-domain questions | 1.00 (4/4) | ≥ 0.75 |
Latency per question on free tiers: median 2.5 s when answered and 1.8 s when refused. The slowest answer took 73.8 s, all of it spent backing off from a free-tier rate limit.
Read these numbers with care:
- 20 questions is a small set. One question moves a rate by 5–6 points.
- The golden set was drafted with an AI assistant from the source texts, then reviewed and approved by me. One reviewer, no inter-annotator agreement.
- Judge scores vary between runs. An earlier local run on the same pipeline code scored faithfulness 0.97; this CI run scored 1.00. The difference is a single verdict on q09, which inspection showed to be a judge false negative (lesson 6d).
- The one false refusal (q13) is a genuine retrieval miss, not a bug: with dense-only retrieval the gold passage never reaches the top 8. Hybrid retrieval fixes it in the local run below.
Hybrid retrieval: before and after
q13 asks how long the Commission has to decide on a request not to apply a TSI. Its answer ("four months … deemed to have been accepted", Directive Art. 7) shares few words with the question, and dense retrieval ranked other "Commission decision" articles higher. Hybrid retrieval adds BM25 keyword search and fuses both rankings with Reciprocal Rank Fusion (RRF), in Azure AI Search and in the Qdrant fallback.
Retrieval only: the same 20 saved queries, deterministic, no LLM (eval/results/retrieval.json, python -m rail_rag.evaluation.retrieval --stores qdrant,azure):
| Dense | Hybrid |
|---|
| Retrieval recall@8, Qdrant | 0.91 (15/16 gold in top 8) | 1.00 (16/16) |
| Retrieval recall@8, Azure AI Search | 0.91 (15/16) | 1.00 (16/16) |
| q13 gold passage rank | not in top 8 | 6 (both stores) |
| Refusal-floor decisions changed | | none of 20 |
End to end, judge-free: one local run of the full pipeline in hybrid mode, compared with the dense CI run above (eval/results/hybrid-local-judge-free.json):
| Metric | Dense (CI) | Hybrid (local) | Δ |
|---|
| Answerable questions answered | 0.94 (15/16) | 1.00 (16/16) | +1 question (q13) |
| Context precision | 0.94 | 1.00 | +0.06 |
| Context recall | 0.89 | 0.95 | +0.06 |
| Retrieval recall@8 | 0.91 | 1.00 | +0.09 |
| Grounding | 1.00 | 1.00 | 0 |
| Unanswerable questions refused | 1.00 | 1.00 | 0 |
여기까지가 README 앞부분입니다. 나머지는 원문에서 읽을 수 있습니다. README 전문 보기 ↗
