Open Seldon
A visual analytics platform that transforms unstructured, multilingual Telegram data from refugee communities into actionable humanitarian intelligence.
What it does
- Scrapes public Telegram channels using the Telegram API
- Translates multilingual messages to English via Claude
- Classifies messages into a humanitarian needs taxonomy (IOM categories)
- Extracts and geocodes locations mentioned in messages
- Stores structured data and semantic embeddings in PostgreSQL + pgvector
- Serves an interactive dashboard: map, category distribution, streaming chatbot, executive summary generator
Stack
| Layer | Technology |
|---|
| Backend API | FastAPI (Python, async) |
| Database | PostgreSQL 16 + pgvector |
| AI/LLM | Claude API (Anthropic) |
| NER / categorization | GLiNER2 (HuggingFace) |
| Embeddings | all-MiniLM-L6-v2 (sentence-transformers) |
| Geocoding | Nominatim (geopy) |
| Frontend | Next.js (TypeScript) |
| Infrastructure | Docker Compose |
Getting started
Prerequisites
- Docker and Docker Compose
- A
.env file (copy from .env.example and fill in values)
- An Anthropic API key
- Telegram API credentials
Run locally
cp .env.example .env
# fill in ANTHROPIC_API_KEY, TELEGRAM_API_ID, TELEGRAM_API_HASH, etc.
docker compose -f docker-compose.yml -f docker-compose.local.yml up --build
The API will be at http://localhost:8000, docs at http://localhost:8000/docs, and the web app at http://localhost:3000.
Run the pipeline
# Scrape a Telegram channel
docker compose exec api python -m pipeline.scraper --channel polska_grupa_informacyjna
# Scrape every enabled channel and process everything waiting (what the daily job runs)
docker compose exec api python -m pipeline.run_daily
Project structure
api/ FastAPI backend + processing pipeline
web/ Next.js frontend
scripts/ Database init, utility scripts
.github/ CI workflows
Contributing
See CODE_OF_CONDUCT.md. Issues and pull requests welcome.
License
MIT — see LICENSE.
