Sports Data Platform
A hybrid cloud sports analytics platform built on a microservices architecture, deployed to a self-hosted Kubernetes cluster with Azure managed services.
Overview
Sports data aggregation and analysis platform that ingests data from multiple external sources (ESPN, CBS Sports, Yahoo!, sportsData.io), transforms it into canonical domain models, and exposes it through a unified API. Currently focused on NCAA Football with planned expansion to NFL, MLB, and PGA. ESPN is the active data provider; CBS, Yahoo!, and sportsData.io integrations are planned.
Key Features
- Multi-source data aggregation - Unified interface for ESPN (active), CBS, Yahoo!, and sportsData.io (planned) via a provider/producer pipeline
- Event-driven architecture - RabbitMQ (local) / Azure Service Bus (production) messaging for decoupled domain communication
- Hybrid cloud deployment - Self-hosted Kubernetes on bare metal, leveraging Azure managed services for specific workloads
- Domain-driven design - Clear bounded contexts with dedicated HTTP clients per domain
- Microservices - Independent services with per-service databases, Dockerfiles, and deployment manifests
- Cost-optimized architecture - Pragmatic mix of on-premises compute and cloud-managed services
Tech Stack
Backend: .NET 10, ASP.NET Core, Entity Framework Core
Infrastructure: Kubernetes (on-prem), Docker, Azure (Service Bus, Cosmos DB, App Configuration, Key Vault, Static Web Apps, DevOps)
Frontend: React (sd-ui), Expo/React Native (sd-mobile), TypeScript
Auth: Firebase Authentication
CI/CD: Azure Pipelines, GitHub Actions, GitOps
Monitoring: OpenTelemetry, Seq, Prometheus
Data: PostgreSQL, Redis, Cosmos DB, Azure Blob Storage
Architecture
Microservices with Domain Client Abstraction
Each domain service runs independently with its own database and API. Cross-service communication uses dedicated HTTP clients (ContestClient, FranchiseClient, VenueClient, etc.) defined in the shared Core library.
Design Principles:
- Clear domain boundaries - Each domain has its own client interface and can be developed independently
- Configuration-driven routing - Client base URLs are configurable per environment
- Development velocity - Shared Core library reduces boilerplate while maintaining service independence
- Pragmatic evolution - Services scale independently based on actual needs, not speculation
Current Implementation
All domain clients currently point to the Producer API. While ContestClient, FranchiseClient, and VenueClient are separate HTTP clients with distinct interfaces, they all resolve to the Producer service's API endpoints via configuration.
Migration path:
- Each client has its own configuration key (e.g.,
ContestClientConfig:ApiUrl, FranchiseClientConfig:ApiUrl)
- Currently all configs point to the same Producer API URL
- When a domain needs independent scaling:
- Deploy that domain as a separate service
- Update the configuration to point to the new service URL
- No code changes required - the abstraction boundary already exists
Data Flow
External APIs (ESPN*) → Provider Service → Azure Blob Storage (raw JSON)
↓
Producer Service → Canonical Domain Models → PostgreSQL
↓
Message Bus (RabbitMQ / Azure Service Bus)
↓
Domain Services (Contest, Franchise, Venue, etc.)
↓
API Gateway → Clients (Web, Mobile)
Project Structure
| Project/Service | Purpose |
|---|
| core | Shared services, components, and middleware consumed by all services |
| api | API Gateway - unified entry point for all client applications. Firebase auth middleware. |
| contest | Games, scores, and statistics. Domain boundary via ContestClient. |
| franchise | Teams, rosters, and metadata. Domain boundary via FranchiseClient. |
| notification | User notifications and alerts. |
| player | Athlete profiles and statistics. |
| producer | Transforms external JSON into canonical domain objects. Publishes integration events via MassTransit. |
| provider | Ingests data from external sources (ESPN active; CBS, Yahoo!, sportsData.io planned). Stores raw JSON in Azure Blob Storage. Schedules sourcing runs via Hangfire. |
Related Repositories
System Diagrams
High-Level Architecture
flowchart TD
PV[Provider]
BLOB[(Blob Storage)]
PV --> BLOB
PD[Producer]
PD <--> BLOB
MSG[RabbitMQ / Azure Service Bus]
N[Notification]
C[Contest]
S[Season]
V[Venue]
PL[Player]
FR[Franchise]
API[API Gateway]
AUTH[Firebase Auth]
API <--> AUTH
WCWEB[Web App]
WCMOB[Mobile App]
PV <--> ESPN[ESPN]
PV -.-> CBS[CBS]
PV -.-> YAHOO[Yahoo!]
PV -.-> SDIO[sportsData.io]
PV <--> MSG
PD <--> MSG
N <--> MSG
C <--> MSG
S <--> MSG
V <--> MSG
PL <--> MSG
FR <--> MSG
API --> C
API --> S
API --> V
API --> PL
API --> FR
API --> N
WCWEB --> API
WCMOB --> API
Detailed Service Architecture
flowchart BT
subgraph Provider
PV[svc]
PVDB[(DB)]
PV-->PVDB
PVAPI[API]
PV-->PVDB
PVAPI-->PVDB
end
BLOB[(Blob Storage)]
PV --> BLOB
subgraph Producer
PD[svc]
PDDB[(DB)]
PDAPI[API]
PD-->PDDB
PDAPI-->PDDB
end
PD <--> BLOB
M[RabbitMQ / Azure Service Bus]
subgraph Notification
N[svc]
NDB[(DB)]
NAPI[API]
end
subgraph Contest
C[svc]
CDB[(DB)]
CAPI[API]
end
subgraph Season
S[svc]
SDB[(DB)]
SAPI[API]
end
subgraph Venue
V[svc]
VDB[(DB)]
VPI[API]
end
subgraph Player
PL[svc]
PLDB[(DB)]
PLAPI[API]
end
subgraph Franchise
FR[svc]
FRDB[(DB)]
FRAPI[API]
end
API[API Gateway]
AUTH[Firebase Auth]
API --> AUTH
WCWEB[Web App]
WCMOB[Mobile App]
WCAPP[Code]
Provider --> ESPN[ESPN]
Provider -.-> CBS[CBS]
Provider -.-> YAHOO[Yahoo!]
Provider -.-> SDIO[sportsData.io]
PV-->M
PD-->M
N-->M
C-->M
S-->M
V-->M
PL-->M
FR-->M
API-->Provider
API-->Contest
API-->Season
API-->Venue
API-->Player
API-->Franchise
API-->Notification
WCWEB-->API
WCMOB-->API
WCAPP-->API
Deployment
Architecture:
- Production - Self-hosted Kubernetes cluster on bare metal
- 4-node cluster: AMD Ryzen 5 7640HS (6-core/12-thread @ 5.0GHz), ~31.5GB usable RAM (BIOS tuned, iGPU allocation minimized), 1TB NVMe PCIe 4.0 SSD
- Cluster totals: 24 cores, 48 threads, ~126GB RAM, 4TB NVMe storage
- Dedicated PostgreSQL node: Same specs, isolated for database workload
- Dual 2.5GbE networking per node
- Azure Managed Services - Service Bus, Cosmos DB, App Configuration, Key Vault, Static Web Apps
- Hybrid approach - On-premises compute, Azure for managed services (born from Azure credit constraints, evolved into pragmatic architecture)
CI/CD Pipeline:
- GitHub Actions - PR validation, automated testing (mobile CI, code review)
- Azure Pipelines - Build, containerization, deployment
- GitOps - Kubernetes manifests managed in sports-data-config
Configuration:
- Azure App Configuration - centralized configuration management
- Azure Key Vault - secrets and sensitive configuration
- Environment-specific overrides via Azure DevOps pipelines
License
This project is licensed under the GNU General Public License v3.0 (GPL-3.0).
Any derivative works must also be open source and distributed under the same GPL-3.0 license. This ensures the community benefits from improvements and contributions.
Note: This is an active development project. Architecture and implementation details are subject to change as requirements evolve.
