KaleMart24 — Multi-Tenant Grocery Management Platform

🌐 Live demo: kalemart.sreekanthp.com

📊 Public Grafana dashboards: grafana.sreekanthp.com (read-only viewer)

A production-grade, multi-tenant SaaS platform for grocery store chains — built to demonstrate full-stack engineering, Kubernetes operations, observability, and autonomous SRE practices.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Kubernetes (OrbStack)                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌───────────────┐  │
│  │ Next.js  │  │ Node.js  │  │  Python  │  │  SRE Agent    │  │
│  │ Frontend │→ │ Backend  │→ │ AI Svc   │  │  (FastAPI +   │  │
│  │ :3000    │  │ :4000    │  │ :5000    │  │   Claude API) │  │
│  └──────────┘  └──────────┘  └──────────┘  └───────────────┘  │
│       │              │              │               ↑            │
│       └──────────────┴──────────────┘        Alertmanager       │
│                      │                             │             │
│               ┌──────────────┐         ┌───────────────────┐   │
│               │  PostgreSQL  │         │    Prometheus +    │   │
│               │  (RLS + RBM) │         │  Grafana + Tempo   │   │
│               └──────────────┘         └───────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Key Features

Multi-Tenancy

• PostgreSQL Row-Level Security (RLS) isolates data per tenant at the DB layer
• Composite indexes (tenant_id, created_at) with explicit query params for optimal planner stats
• SET LOCAL random_page_cost = 1.1 per transaction — corrects PostgreSQL's cost model for SSD storage
• Tax period queries: 145–255 ms (down from 22,000 ms before RLS-aware index tuning)

AI Supervisor

• Claude-powered inventory analysis (/api/supervisor/analyze)
• Identifies low-stock, reorder recommendations, and sales velocity trends
• Mode falls back gracefully: claude → rule-based → demo

Autonomous SRE Agent

The centrepiece for SRE showcase. A Claude-powered agent that acts as an on-call SRE:

Prometheus alert fires
        ↓
  Alertmanager webhook → SRE Agent (FastAPI)
        ↓
  Claude investigates using tools:
    • query_prometheus   — fetch metrics
    • get_pod_logs       — stream k8s logs
    • get_k8s_events     — cluster events
    • run_db_diagnostic  — EXPLAIN ANALYZE (read-only guard)
    • restart_deployment — needs human approval
    • create_db_index    — needs human approval
        ↓
  Slack notification with Approve / Reject / Don't-ask-again links
        ↓
  Fix applied → metrics re-validated → RCA posted to Slack

Approval flow: Uses asyncio.Event shared between the background investigation task and the HTTP /approve/{id} endpoint. Zero polling — the task suspends until the human clicks a link.

Don't-ask-again: Approved action types are persisted so repeated incidents self-heal without paging the engineer.

Observability Stack

• Prometheus — scrapes backend, AI service, and OTel Collector metrics
• Grafana — public read-only dashboards at grafana.sreekanthp.com
• Tempo — distributed traces via OTel SDK (Node.js + Python)
• Alert rules — latency, errors, pods, infrastructure, and SLO burn rate
• SLO burn rate alerts — fast-burn (1h window, 10× budget) fires in ≤ 5 minutes

Recruiter access is intentionally read-only: production Grafana enables anonymous Viewer access, loads the Kalemart dashboard ConfigMap through Argo CD, and is exposed through Cloudflare Tunnel at https://grafana.sreekanthp.com.

Tech Stack

Layer	Technology
Frontend	Next.js 14, React, Tailwind CSS
Backend	Node.js, Express, OpenTelemetry
AI Service	Python, FastAPI, Anthropic SDK
SRE Agent	Python, FastAPI, Anthropic Claude claude-sonnet-4-6
Database	PostgreSQL 15, Row-Level Security
Infra	Kubernetes, Helm, Docker
Observability	Prometheus, Alertmanager, Grafana, Tempo, OTel Collector
CI/CD	GitHub Actions — build & push multi-service images

SRE Practices Demonstrated

• SLO / Error Budget — /api 99.5% availability SLO with burn rate alerting
• Agentic incident response — LLM-driven investigate → diagnose → fix → validate loop
• Human-in-the-loop — destructive actions require approval; safe reads run automatically
• Toil reduction — "don't ask again" auto-approves repeated safe remediations
• Query performance tuning — EXPLAIN ANALYZE, composite indexes, cost model correction
• Distributed tracing — trace IDs flow from frontend → backend → AI service
• Production smoke checks — live demo login, inventory API, Ops page, and public Grafana are verified after deploys

Quick Start (Local Kubernetes)

Prerequisites

• OrbStack or any Kubernetes (kind, minikube)
• kubectl, helm, docker

1. Clone & configure

git clone https://github.com/sreekanthca6/kalemart.git
cd kalemart
cp .env.example .env
# Fill in .env — at minimum: ANTHROPIC_API_KEY, PGPASSWORD, JWT_SECRET

2. Build images

docker build -t kalemart/backend:latest    ./apps/backend
docker build -t kalemart/frontend:latest   ./apps/frontend
docker build -t kalemart/ai-service:latest ./apps/ai-service
docker build -t kalemart/sre-agent:latest  ./apps/sre-agent

3. Deploy

helm install kalemart ./helm/kalemart \
  --set secrets.anthropicApiKey="$ANTHROPIC_API_KEY" \
  --set secrets.postgresPassword="$PGPASSWORD" \
  --set secrets.jwtSecret="$JWT_SECRET" \
  --set secrets.slackWebhookUrl="$SLACK_WEBHOOK_URL"

4. Install observability stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Slack notifications for Prometheus alerts.
# The webhook URL is mounted into Alertmanager from a Secret, never committed to Git.
kubectl create namespace observability --dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic alertmanager-slack-webhook \
  -n observability \
  --from-literal=url="$SLACK_WEBHOOK_URL" \
  --dry-run=client -o yaml | kubectl apply -f -

helm install prometheus prometheus-community/prometheus \
  -n observability --create-namespace \
  -f observability/prometheus-values.yaml

kubectl apply -f observability/alert-rules.yaml

For local OrbStack testing, ./scripts/deploy-local.sh dev uses observability/helmfile.local.yaml so Grafana points to localhost instead of production:

# Required if you want to test Slack alert delivery locally.
export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/..."
kubectl create namespace observability --dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic alertmanager-slack-webhook \
  -n observability \
  --from-literal=url="$SLACK_WEBHOOK_URL" \
  --dry-run=client -o yaml | kubectl apply -f -

./scripts/deploy-local.sh dev

5. Port-forward & explore

./scripts/port-forward.sh
# Frontend:     http://localhost:3000
# Grafana:      http://localhost:3001
# Prometheus:   http://localhost:9090
# Alertmanager: http://localhost:9093

6. Explore the SRE showcase

Open http://localhost:3000/ops to see the recruiter-facing control room. From there, open Grafana dashboards for SRE overview, API latency, platform infrastructure, Prometheus alerts, AI service health, and inventory business metrics.

Project Structure

kalemart/
├── apps/
│   ├── backend/          # Node.js/Express API
│   ├── frontend/         # Next.js dashboard
│   ├── ai-service/       # Python FastAPI — Claude AI supervisor
│   └── sre-agent/        # Python FastAPI — autonomous SRE agent
├── helm/kalemart/        # Helm chart for all services
├── observability/
│   ├── prometheus-values.yaml
│   └── alert-rules.yaml  # 10 Prometheus alert rules
├── scripts/              # DB seed & Shopify import
└── .github/workflows/    # CI — build & push images

CI/CD

GitHub Actions builds and pushes all 4 Docker images on every push to main. See .github/workflows/.

License

MIT