LLMstats

Real-time benchmarking of free-tier models from OpenRouter, Groq, and Mistral — automated roughly every 3 hours, zero infrastructure.

🌐 saif658.github.io/LLMstats — live, public, refreshes roughly every 3 hours

Forked from the architecture of NIMStats, rebuilt to compare three OpenAI-compatible providers side-by-side.

---

✨ What is LLMstats?

LLMstats automatically benchmarks 39 free-tier models roughly every 3 hours — 8 served by OpenRouter (the unified router, 5 upstream providers), 5 by Groq (the LPU cloud), and 26 by Mistral (Experiment tier) — and publishes the results to a live, interactive dashboard. No servers, no subscriptions — just fork the repo, drop in three API keys, and go.

🏎️ Every ~3h	📊 Interactive Charts	🔁 Zero Infra	🔀 Multi-Provider
GitHub Actions cron	Speed, throughput, reliability	Static site + free CI/CD	OpenRouter ↔ Groq ↔ Mistral comparisons

---

⚡ Quick Start

A working dashboard in under 5 minutes.

1. Fork & clone

git clone https://github.com/Saif658/LLMstats.git
cd LLMstats

2. Get free API keys

Provider	Where to get it
OpenRouter	openrouter.ai/keys — sign up, generate a key
Groq	console.groq.com/keys — sign up, generate a key
Mistral	console.mistral.ai/api-keys — sign up, generate a key (La Plateforme / Experiment tier)

3. Add secrets

In your forked repo: Settings → Secrets and variables → Actions → New repository secret

Name	Value
OPENROUTER_API_KEY	Your OpenRouter API key
GROQ_API_KEY	Your Groq API key
MISTRAL_API_KEY	Your Mistral API key

4. Enable GitHub Pages

Settings → Pages → Build from "GitHub Actions".

5. Run your first benchmark

Actions → Benchmark OpenRouter + Groq + Mistral Models → Run workflow.

The dashboard auto-refreshes roughly every 3 hours after that. ✨

---

📊 Dashboard

Tab	What you get
📊 Overview	5 KPI cards · success-rate trend · top-10 speed & throughput bars · per-model reliability pills
🏆 Leaderboard	Composite score · sortable columns · trend indicators · provider chips (OpenRouter vs Groq vs Mistral)
🔬 Explorer	Per-model deep dive · response time history · error breakdown · availability heatmap
⏱ Timeline	Filterable run history · expandable run cards with full per-model detail
⚔️ Compare	Head-to-head overlay · win-rate stats · side-by-side metric comparison

---

🤖 Benchmarked models (free tier)

🟣 OpenRouter · 8 models · 5 upstream providers

Upstream	Model	Note
NVIDIA	nvidia/nemotron-3-ultra-550b-a55b:free	550B MoE flagship
NVIDIA	nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free	Compact reasoning
NVIDIA	nvidia/nemotron-3.5-super-vl:free	New model
Google	google/gemma-4-26b-a4b-it:free	Gemma 4 mid-size
Google	google/gemma-4-31b-it:free	New model
Cohere	cohere/north-mini-code:free	Cohere North mini-code
liquid	liquid/lfm-3-1.7b:free	New model
recursive	recursive/flint-v1.0:free	New model

🟠 Groq · 5 chat models

Model	Note
openai/gpt-oss-120b	Open-source 120B
openai/gpt-oss-20b	Open-source 20B
openai/gpt-oss-safeguard-20b	Safety-guard 20B
allam-2-7b	Allam (small MoE)
qwen/qwen3.8-27b	New model

🔵 Mistral · 26 chat models (Experiment tier)

Model	Note
mistral-medium-latest	Mistral Medium
mistral-small-latest	Mistral Small
ministral-8b-latest	Ministral 8B
ministral-3b-latest	Ministral 3B
magistral-small-latest	Magistral small
codestral-latest	Code-specialized
codestral-2508	New model
labs-leanstral-1-5	New model
labs-leanstral-1-5-1	New model
ministral-14b-2512	New model
ministral-14b-latest	New model
ministral-3b-2512	New model
ministral-8b-2512	New model
mistral-code-latest	New model
mistral-medium	New model
mistral-medium-2604	New model
mistral-medium-3	New model
mistral-medium-3-5	New model
mistral-medium-3.5	New model
mistral-small-2603	New model
mistral-vibe-cli-fast	New model
mistral-vibe-cli-latest	New model
mistral-vibe-cli-with-tools	New model
magistral-medium-latest	Magistral medium
mistral-large-4	New model
mistral-large-4-0	New model

---

🏗️ How it works

┌─── GitHub Actions (every ~3h) ────────────────────────────────────────────┐
│                                                                          │
│   ┌─────────────────────┐         ┌─────────────────────┐                │
│   │  Job 1 — Group A    │         │  Job 2 — Group B    │  in parallel    │
│   │  39 models (mixed)  │         │  34 models (mixed)  │                │
│   └──────────┬──────────┘         └──────────┬──────────┘                │
│              └──────────────┬───────────────┘                           │
│                     ┌────────▼────────┐                                 │
│                     │  Merge + commit │  → benchmark/history.db + docs/history.db │
│                     └─────────────────┘                                 │
└─────────────────────────────────────────────────────────────────────────┘
                                      │
                          ┌───────────▼──────────┐
                          │  Static Dashboard    │  GitHub Pages
                          │  (sql.js client)     │
                          └──────────────────────┘

Parallel matrix ≈ 50% faster ⚡

---

🛠️ Customization

Edit PROMPT in benchmark/test_models.py:

PROMPT = "Your custom prompt here"

Edit OPENROUTER_FREE_MODELS, GROQ_MODELS, or MISTRAL_MODELS in benchmark/test_models.py. The lists get split roughly in half between group1 and group2 jobs.

Edit .github/workflows/benchmark.yml:

- cron: '0 */3 * * *'   # every 3 hours (default)
- cron: '0 */1 * * *'   # every hour
- cron: '0 0 */6 * *'   # every 6 hours

# Serve the dashboard from the repo root (serves docs/index.html + docs/history.db)
python3 -m http.server 8000
# Open http://localhost:8000/docs/

# Run benchmarks manually (requires API keys as env vars)
export OPENROUTER_API_KEY=...
export GROQ_API_KEY=...
export MISTRAL_API_KEY=...
python3 benchmark/test_models.py            # all models
MODEL_GROUP=group1 python3 benchmark/test_models.py   # one half

---

📦 Data storage

benchmark/history.db is a SQLite database persisted in the repo. The browser loads it via sql.js (WebAssembly) and queries it entirely client-side. benchmark/results*.json are temporary per-job artifacts that never get committed (.gitignored). On every merge the workflow mirrors the latest benchmark/history.db to docs/history.db next to the dashboard.

Schema:

runs          (id, timestamp, prompt, success_count, total_models, fastest_model, fastest_time)
model_results (run_id, model, provider, success, error, response_time, tokens_generated, total_tokens, response)

Results are pruned to the most recent 720 runs to keep history.db from growing unbounded.

Benchmark parameters: temperature: 0.7 · top_p: 0.9 · max_tokens: 500 · OpenAI-compatible chat completions API.

---

🤝 Contributing

PRs and ideas welcome:

• 🆕 Add new free-tier models from each provider
• 📊 New chart types or dashboard widgets
• 🐛 Bug fixes and perf improvements
• 📖 Docs / translations

---

---

Inspired by NIMStats · ⚡ Powered by GitHub Actions