🎯 What is StackRadar?
StackRadar re-reads the developer ecosystem every 30 minutes:
- ⭐ GitHub — stars, forks, open issues and the latest release (authenticated API, ETag caching, rate-limit handling)
- 🟠 Hacker News — the front page, plus a search for each tool over the last 48 hours
- 📝 Dev.to — the day's 100 top articles
- 🔴 Reddit — 35 developer subreddits, read as RSS in four combined requests
- 📰 Tech news — ten tech and developer RSS feeds
- 🤖 Sentiment — a Groq-hosted LLM classifies community sentiment per tool. It is displayed; it is not an input to the score.
- 💼 Hiring demand — how many recent Ask HN "Who is hiring?" posts name each tool (Jul–Sep 2026: 756 posts, 24 of 31 tools named). Shown beside the score, always with its sample size; never part of the score.
Each tool gets a 0–100 momentum score from its own signals only: log-normalised GitHub stars (60%) and forks (15%), and community mentions across the four discussion sources (25%). The scale is absolute, not a percentile rank, so a tool's score does not move when an unrelated tool moves. Trend, growth stage and the recommendation text come from fixed rules and templates in scoring.py, and the learning-priority rating puts measured hiring demand before trend.
🏗️ Architecture
┌──────────────────────────────────────────────┐
│ Frontend │
│ Next.js 16 + Tailwind + Recharts + Clerk │
│ ISR Caching (30 min) │
│ :3000 │
└─────────────────┬────────────────────────────┘
│ REST API
┌─────────────────▼────────────────────────────┐
│ Backend │
│ FastAPI + SQLAlchemy + slowapi + loguru │
│ :8000 │
│ ┌──────────┐ ┌────────────┐ ┌────────────┐ │
│ │ Scraper │ │ Scoring │ │ Sentiment │ │
│ │ (30min) │ │ Engine │ │ (Groq LLM) │ │
│ └──────────┘ └────────────┘ └────────────┘ │
└─────────────────┬────────────────────────────┘
│
┌─────────────────▼────────────────────────────┐
│ PostgreSQL (Docker) / SQLite (Dev) │
└──────────────────────────────────────────────┘
🛡️ Hackathon Architecture: Agent Reliability & Security
Produced for the MOSS Zero-Latency Builder Sprint — track Agent Reliability,
Security and Evaluation. Scoped honestly: the runtime guardrail and the
grounding half of Context Eval are a working prototype in this repo
(app/services/guardrails.py); the named third-party stack — Enkrypt AI
(guardrails), Qdrant (gRPC retrieval), Mastra (agent evals + tracing) — is
designed in, not yet wired. Each row below is tagged with its real status.
Where the LLM actually sits today
StackRadar is mostly deterministic. Scoring (app/services/scoring.py) is pure
arithmetic over GitHub + community signals — no model involved. The only LLM
call on the live path is a single Groq request per scrape cycle that classifies
community sentiment (batch_sentiment_analysis in app/services/scraper.py), and it
is a no-op when GROQ_API_KEY is unset. There is no agent, no retrieval-augmented
generation, and no user-facing text generation yet.
That single call is the seam this layer wraps, and the seam any future generative
feature (e.g. an "explain this trend" agent) would pass through.
The "Trust Loop"
| Component | Tech (prototype → production target) | Status | Role |
|---|
| Runtime Guardrails | Pydantic v2 → Enkrypt AI | ✅ prototype in repo | Every model response is validated against a strict schema (SentimentVerdict) before any sentiment reaches scoring.py. Malformed rows are dropped, a fully unparseable response rejects the whole batch, and a verdict whose index is outside the batch (a hallucinated index) is discarded. Anything not explicitly accepted stays at the safe neutral default. Enkrypt AI would add hallucination / safety / PII detectors on the same seam. |
| Context Eval (grounding) | Python scoring regex → Mastra evals | ✅ prototype in repo | A non-neutral verdict is kept only if the source headline actually mentions a tracked tool (scoring.classify_text_to_tools). Sentiment that can't be grounded in our domain is quarantined to neutral rather than persisted. Faithfulness scoring against retrieved context is the proposed extension. |
| Fast Retrieval | — → Qdrant (gRPC) | ○ designed, not wired | Low-latency vector retrieval — would serve grounding snippets (tool docs, prior snapshots) to the LLM call over gRPC instead of dumping raw scrape text into the prompt, and back the faithfulness score above. |
| Latency Tracing | guardrails.traced() → OpenTelemetry / Mastra | ◑ stub in repo | Wraps the inference and validation hops and logs span durations today; swapping in a real exporter leaves the call sites unchanged. |
flowchart LR
subgraph current["Scrape pipeline"]
SC[Scheduler loop - 30 min]
SCR[Scraper - GitHub, HN, Reddit, Dev.to, RSS]
GROQ[Groq LLM - sentiment classification]
SCORE[Scoring engine - deterministic, absolute 0-100]
DB[(PostgreSQL / SQLite)]
SC --> SCR --> GROQ
SCORE --> DB
end
subgraph trustloop["Trust Loop"]
GUARD[Runtime Guardrails - Pydantic schema + index check - SHIPPED - Enkrypt AI proposed]
EVAL[Context Eval - grounding check SHIPPED - faithfulness / Mastra evals proposed]
QDRANT[Qdrant - gRPC vector retrieval - PROPOSED]
TRACE[Latency Tracing - traced stub now - OpenTelemetry proposed]
end
QDRANT -. retrieved snippets .-> GROQ
GROQ -- raw response --> GUARD
GUARD -- validated --> EVAL
EVAL -- accepted --> SCORE
EVAL -- ungrounded, forced neutral --> SCORE
TRACE -. spans .-> GROQ
TRACE -. spans .-> GUARD
The loop: Qdrant (gRPC retrieval) → LLM → runtime guardrails (schema + index) → Context Eval (grounding) → persist or quarantine, with tracing around every hop.
Nothing the LLM produces is written to the database until it has passed the guardrail
schema check and cleared the grounding check. Offline proof:
backend/scripts/check_guardrails.py runs the guardrail against well-formed,
malformed, hallucinated-index and ungrounded responses with no network or API key.
Full requirements, milestones and open risks: docs/architecture/PRD.md.
A conceptual diagram from the sprint design tool is kept at
docs/architecture/proposed-reliability-layer.pdf
(PNG). Its "Existing Infrastructure"
boxes are a generic template and differ from the real stack (e.g. the collector is Python,
not Node.js; there is no Kong gateway) — the Mermaid diagram above is the accurate
current-state reference.
📸 Pages
| Page | Description |
|---|
Dashboard (/) | Tool cards with scores, search bar, category filters, bookmark buttons |
Trends (/trends) | Bar chart of all tool scores + compact cards |
Compare (/compare) | Side-by-side comparison of 2–5 tools with metrics table + history overlay chart |
Explore (/explore) | Domain-level view with expandable tool lists |
Roadmaps (/roadmaps) | Curated learning roadmaps (e.g. "Become a DevOps Engineer") |
Tool Detail (/tools/[slug]) | Deep-dive with history chart, sentiment, GitHub stats, SEO metadata |
Watchlist (/watchlist) | Personal bookmarked tools (requires sign-in via Clerk) |
🚀 Quick Start
Option 1: Docker (Recommended)
# Clone the repo
git clone https://github.com/amantebriwal4321/StackRadar.git
cd StackRadar
# Add your API keys
cp backend/.env.example backend/.env
cp frontend/.env.example frontend/.env.local
# Edit both files with your actual keys
# Start everything
docker-compose up --build
Option 2: Local Development
Backend:
cd backend
python -m venv venv
.\venv\Scripts\Activate.ps1 # Windows
# source venv/bin/activate # Mac/Linux
pip install -r requirements.txt
# Create .env file with your keys (see Environment Variables below)
python -m uvicorn app.main:app --host 0.0.0.0 --port 8000
Frontend:
cd frontend
npm install
# Create .env.local with your Clerk keys (see Environment Variables below)
npm run dev
🔑 Environment Variables
Backend (backend/.env)
| Variable | Required | Description |
|---|
GITHUB_TOKEN | ✅ | GitHub Personal Access Token (classic, repo:read scope) |
GROQ_API_KEY | ✅ | Groq API key for LLM sentiment analysis |
DATABASE_URL | ❌ | PostgreSQL URL (auto-set by Docker; defaults to SQLite) |
ADMIN_API_KEY | ❌ | Secret key for POST /admin/scrape manual trigger |
Frontend (frontend/.env.local)
| Variable | Required | Description |
|---|
BACKEND_ORIGIN | ❌ | Where the Next server proxies /api/v1/* (read at build time; defaults to http://localhost:8000, docker-compose sets http://backend:8000) |
NEXT_PUBLIC_SITE_URL | ❌ | Absolute site URL for OG/canonical/sitemap |
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY | ✅ | Clerk publishable key from dashboard.clerk.com |
CLERK_SECRET_KEY | ✅ | Clerk secret key |
📡 API Reference
| Method | Endpoint | Description |
|---|
GET | /api/v1/tools | All tools (paginated: ?page=1&per_page=20&category=AI / ML) |
GET | /api/v1/tools/compare?slugs=react,rust | Compare 2–5 tools side by side |
GET | /api/v1/tools/{slug} | Tool detail with decision intelligence |
GET | /api/v1/tools/{slug}/history?days=30 | Time-series score data (30/90/365 days) |
GET | /api/v1/tools/{slug}/resources | Best free videos and platform links for a tool (ranked by reach, engagement, freshness; verified live) |
GET | /api/v1/tools/{slug}/projects | Buildable project briefs for a tool |
GET | /api/v1/domains | Domain-level summaries |
GET | /api/v1/domains/{slug}/learning-path | Ordered learning path for a domain |
GET | /api/v1/roadmaps | All learning roadmaps |
GET | /api/v1/roadmaps/{slug} | Full roadmap: steps, tools, projects, and a career brief with measured hiring counts for the focus domains |
GET | /api/v1/projects | Project briefs, filterable by tool, domain (category or roadmap slug) and tier |
GET | /api/v1/projects/{slug} | One brief with its step-by-step walkthrough, docs and a verified video |
GET | /api/v1/overview | Headline figures and the time of the last successful scrape |
GET | /api/v1/status | Scraper status: current step, per-source counts, consecutive failures, recent errors |
GET | /api/v1/health |
🛠️ Tech Stack
| Layer | Technology |
|---|
| Frontend | Next.js 16, React 19, TypeScript, Tailwind CSS 4, Recharts, Framer Motion, Lucide Icons |
| Auth | Clerk (@clerk/nextjs) — Google, email, social login |
| Caching | ISR (Incremental Static Regeneration) — 30-minute revalidation |
| Backend | Python 3.10, FastAPI, SQLAlchemy 2.x, Alembic, Pydantic v2 |
| Logging | Loguru (structured, rotating file logs, JSON in production) |
| Security | slowapi rate limiting (60 req/min per IP), CORS hardening |
| AI/ML | Groq LLM (Llama 3.3 70B) for batch sentiment analysis |
| Data Sources | GitHub API, HackerNews Firebase API, Dev.to API, Reddit RSS, News RSS |
| Database | PostgreSQL 15 (production, pooled connections), SQLite (development) |
| Infrastructure | Docker, Docker Compose, Kubernetes manifests, GitHub Actions CI |
| SEO | Dynamic generateMetadata(), sitemap.xml, robots.txt |
📂 Project Structure
StackRadar/
├── backend/
│ ├── app/
│ │ ├── api/endpoints/mvp.py # All API routes (tools, domains, roadmaps, health)
│ │ ├── core/config.py # Pydantic settings & env vars
│ │ ├── db/ # SQLAlchemy session, base, connection pooling
│ │ ├── models/all_models.py # Tool, Domain, ToolSnapshot, Roadmap models
│ │ └── services/
│ │ ├── scheduler.py # Background scraper loop (30-min cycle)
│ │ ├── scraper.py # GitHub, HN, Dev.to, Reddit, News fetchers
│ │ ├── scoring.py # TOOL_REGISTRY + logarithmic scoring engine
│ │ └── seed.py # Initial data seeding (tools, domains, roadmaps)
│ ├── logs/ # Rotating log files (loguru)
│ ├── alembic/ # Database migrations
│ ├── Dockerfile
│ └── requirements.txt
├── frontend/
│ ├── src/
│ │ ├── app/ # Next.js pages (/, /trends, /compare, /watchlist, etc.)
│ │ │ ├── sitemap.ts # Dynamic sitemap for SEO
│ │ │ ├── robots.ts # robots.txt configuration
│ │ │ └── watchlist/ # User watchlist (Clerk auth required)
│ │ ├── components/ # Navbar, TrendCard, WatchlistButton, ChartContainer
│ │ └── data/trends.ts # API client & TypeScript types (ISR cached)
│ ├── Dockerfile
│ └── package.json
├── infrastructure/
│ └── kubernetes/deployment.yaml # K8s deployment with health/readiness probes
├── .github/workflows/
│ ├── backend.yml # Backend CI (lint + Docker build)
│ └── frontend.yml # Frontend CI (TypeScript + build)
├── docker-compose.yml
├── CHANGELOG.md
└── changelog_2026-05-16.md # Detailed phase-by-phase changelog
☁️ Deployment & known limits
The live beta runs the frontend on Vercel and the backend on Render's free web tier, with a Neon Postgres database. Both free tiers have limits that have already shown up in production, and the app is built to report them rather than hide them:
| Limit | What happened | How you can tell |
|---|
| Render free tier spins down after ~15 min idle | The first request after a quiet spell takes ~35 s (measured; see the header of keep-awake.yml) | A scheduled workflow pings the backend to keep it warm; it costs most of the free instance-hours, which its comments say plainly |
| Neon free-tier quota | By 30 Sep 2026 the database was refusing connections ("exceeded the quota") and the scraper had failed 14 cycles in a row. It was succeeding again by 3 Oct | GET /api/v1/status lists the error and the failure count; GET /api/v1/health/data returns 503, not a crash, when the database is unreachable (covered by a test) |
If you run your own instance, point an uptime monitor at /api/v1/health/data: it is the one endpoint built to answer "is the data current?", and /api/v1/health deliberately stays 200 so a platform health check never restarts the process over stale data. Setup is in DEPLOY.md.
🧪 Development
# Run backend with hot reload
cd backend && python -m uvicorn app.main:app --reload
# Backend tests: ~190, ~2s, no database, keys or network needed
cd backend && pip install -r requirements-dev.txt && python -m pytest tests -q
# Backend lint + format (CI enforces both)
cd backend && ruff check app/ && ruff format --check app/
# Production build check (frontend)
cd frontend && npm run build
# Type-check frontend
cd frontend && npx tsc --noEmit
# Run Alembic migrations
cd backend
alembic revision --autogenerate -m "description"
alembic upgrade head
📝 Changelog
See CHANGELOG.md for the original development history, and changelog_2026-05-16.md for the Phase 1-5 production upgrade changelog.
📜 License
MIT — see LICENSE for details.
