LLMstats
Real-time benchmarking of free-tier models from OpenRouter, Groq, and Mistral — automated roughly every 3 hours, zero infrastructure.
🌐 saif658.github.io/LLMstats — live, public, refreshes roughly every 3 hours
Forked from the architecture of NIMStats, rebuilt to compare three OpenAI-compatible providers side-by-side.
✨ What is LLMstats?
LLMstats automatically benchmarks 39 free-tier models roughly every 3 hours — 8 served by OpenRouter (the unified router, 5 upstream providers), 5 by Groq (the LPU cloud), and 26 by Mistral (Experiment tier) — and publishes the results to a live, interactive dashboard. No servers, no subscriptions — just fork the repo, drop in three API keys, and go.
| 🏎️ Every ~3h | 📊 Interactive Charts | 🔁 Zero Infra | 🔀 Multi-Provider |
|---|
| GitHub Actions cron | Speed, throughput, reliability | Static site + free CI/CD | OpenRouter ↔ Groq ↔ Mistral comparisons |
⚡ Quick Start
A working dashboard in under 5 minutes.
1. Fork & clone
git clone https://github.com/Saif658/LLMstats.git
cd LLMstats
2. Get free API keys
3. Add secrets
In your forked repo: Settings → Secrets and variables → Actions → New repository secret
| Name | Value |
|---|
OPENROUTER_API_KEY | Your OpenRouter API key |
GROQ_API_KEY | Your Groq API key |
MISTRAL_API_KEY | Your Mistral API key |
4. Enable GitHub Pages
Settings → Pages → Build from "GitHub Actions".
5. Run your first benchmark
Actions → Benchmark OpenRouter + Groq + Mistral Models → Run workflow.
The dashboard auto-refreshes roughly every 3 hours after that. ✨
📊 Dashboard
| Tab | What you get |
|---|
| 📊 Overview | 5 KPI cards · success-rate trend · top-10 speed & throughput bars · per-model reliability pills |
| 🏆 Leaderboard | Composite score · sortable columns · trend indicators · provider chips (OpenRouter vs Groq vs Mistral) |
| 🔬 Explorer | Per-model deep dive · response time history · error breakdown · availability heatmap |
| ⏱ Timeline | Filterable run history · expandable run cards with full per-model detail |
| ⚔️ Compare | Head-to-head overlay · win-rate stats · side-by-side metric comparison |
🤖 Benchmarked models (free tier)
🟣 OpenRouter · 8 models · 5 upstream providers
| Upstream | Model | Note |
|---|
| NVIDIA | nvidia/nemotron-3-ultra-550b-a55b:free | 550B MoE flagship |
| NVIDIA | nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free | Compact reasoning |
| NVIDIA | nvidia/nemotron-3.5-super-vl:free | New model |
| Google | google/gemma-4-26b-a4b-it:free | Gemma 4 mid-size |
| Google | google/gemma-4-31b-it:free | New model |
| Cohere | cohere/north-mini-code:free | Cohere North mini-code |
| liquid | liquid/lfm-3-1.7b:free | New model |
| recursive | recursive/flint-v1.0:free | New model |
🟠 Groq · 5 chat models
| Model | Note |
|---|
openai/gpt-oss-120b | Open-source 120B |
openai/gpt-oss-20b | Open-source 20B |
openai/gpt-oss-safeguard-20b | Safety-guard 20B |
allam-2-7b | Allam (small MoE) |
qwen/qwen3.8-27b | New model |
🔵 Mistral · 26 chat models (Experiment tier)
| Model | Note |
|---|
mistral-medium-latest | Mistral Medium |
mistral-small-latest | Mistral Small |
ministral-8b-latest | Ministral 8B |
ministral-3b-latest | Ministral 3B |
magistral-small-latest | Magistral small |
codestral-latest | Code-specialized |
codestral-2508 | New model |
labs-leanstral-1-5 | New model |
labs-leanstral-1-5-1 | New model |
ministral-14b-2512 | New model |
ministral-14b-latest | New model |
ministral-3b-2512 | New model |
ministral-8b-2512 | New model |
mistral-code-latest | New model |
mistral-medium | New model |
mistral-medium-2604 | New model |
mistral-medium-3 | New model |
mistral-medium-3-5 | New model |
mistral-medium-3.5 | New model |
mistral-small-2603 | New model |
mistral-vibe-cli-fast | New model |
mistral-vibe-cli-latest | New model |
mistral-vibe-cli-with-tools | New model |
magistral-medium-latest | Magistral medium |
mistral-large-4 | New model |
mistral-large-4-0 | New model |
🏗️ How it works
┌─── GitHub Actions (every ~3h) ────────────────────────────────────────────┐
│ │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ Job 1 — Group A │ │ Job 2 — Group B │ in parallel │
│ │ 39 models (mixed) │ │ 34 models (mixed) │ │
│ └──────────┬──────────┘ └──────────┬──────────┘ │
│ └──────────────┬───────────────┘ │
│ ┌────────▼────────┐ │
│ │ Merge + commit │ → benchmark/history.db + docs/history.db │
│ └─────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
│
┌───────────▼──────────┐
│ Static Dashboard │ GitHub Pages
│ (sql.js client) │
└──────────────────────┘
Parallel matrix ≈ 50% faster ⚡
🛠️ Customization
Edit PROMPT in benchmark/test_models.py:
PROMPT = "Your custom prompt here"
Edit OPENROUTER_FREE_MODELS, GROQ_MODELS, or MISTRAL_MODELS in benchmark/test_models.py. The lists get split roughly in half between group1 and group2 jobs.
Edit .github/workflows/benchmark.yml:
- cron: '0 */3 * * *' # every 3 hours (default)
- cron: '0 */1 * * *' # every hour
- cron: '0 0 */6 * *' # every 6 hours
# Serve the dashboard from the repo root (serves docs/index.html + docs/history.db)
python3 -m http.server 8000
# Open http://localhost:8000/docs/
# Run benchmarks manually (requires API keys as env vars)
export OPENROUTER_API_KEY=...
export GROQ_API_KEY=...
export MISTRAL_API_KEY=...
python3 benchmark/test_models.py # all models
MODEL_GROUP=group1 python3 benchmark/test_models.py # one half
📦 Data storage
benchmark/history.db is a SQLite database persisted in the repo. The browser loads it via sql.js (WebAssembly) and queries it entirely client-side. benchmark/results*.json are temporary per-job artifacts that never get committed (.gitignored). On every merge the workflow mirrors the latest benchmark/history.db to docs/history.db next to the dashboard.
Schema:
runs (id, timestamp, prompt, success_count, total_models, fastest_model, fastest_time)
model_results (run_id, model, provider, success, error, response_time, tokens_generated, total_tokens, response)
Results are pruned to the most recent 720 runs to keep history.db from growing unbounded.
Benchmark parameters: temperature: 0.7 · top_p: 0.9 · max_tokens: 500 · OpenAI-compatible chat completions API.
🤝 Contributing
PRs and ideas welcome:
- 🆕 Add new free-tier models from each provider
- 📊 New chart types or dashboard widgets
- 🐛 Bug fixes and perf improvements
- 📖 Docs / translations
Inspired by NIMStats · ⚡ Powered by GitHub Actions
