Hermes Model Leaderboard

A LLM comparison dashboard plugin for Hermes Desktop: compare, score and rank LLM models as a full-page /model-leaderboard route — prices, benchmarks and OpenRouter news, no configuration needed.

Ported from the reference app sumaris-agent/docker/llm-dashboard (live at http://localhost:3004/), first tab only: authentication, dark-mode toggle and the consumption tab are intentionally out of scope — Hermes Desktop already provides the identity (the plugin runs inside the authenticated desktop context) and the theme (via the standard CSS variables).

Install

hermes plugins install e-is/hermes-model-leaderboard --force
hermes plugins enable model-leaderboard
hermes gateway restart

Then reload the desktop window (F5 / Cmd+R): Model Leaderboard appears in the sidebar.

Notes — the three things that trip people up:

• --force is expected. The install scanner rates any community plugin CAUTION (bundled React/recharts regexes read as "obfuscation", localStorage as "persistence"). The verdict is CAUTION, not DANGEROUS, so --force applies — there is no CRITICAL finding to accept. Two findings were real enough to fix in the build instead: a SECRET_DO_NOT_PASS_THIS_OR_YOU_WILL_BE_FIRED string inside bundled prop-types (now split by scripts/build.mjs) and a zero-width joiner in an emoji.
• The plugin key is model-leaderboard (from plugin.yaml), not the repository name hermes-model-leaderboard. hermes plugins enable hermes-model-leaderboard will not match. Likewise dashboard/manifest.json must carry the same name: plugin.yaml decides the namespace the renderer's ctx.rest scopes to, while the manifest decides the path the gateway mounts the router under. A mismatch 404s every API call (No such API endpoint: /api/plugins/model-leaderboard/...).
• The desktop needs its own toggle. A plugin coming from ~/.hermes/plugins/ ships disabled in Hermes Desktop until you switch it on in Settings → Plugins. That choice is stored per renderer id: if you enabled an earlier build published as hermes-model-leaderboard, the model-leaderboard renderer is a new id and starts off again — no sidebar entry, no page. Enable it once more.
• enable asks about built-in tool override — "Allow this plugin to replace built-in tools (e.g. shell_exec, write_file)?". Answer no: this plugin registers no agent tools at all (it ships a dashboard API router, a desktop renderer and a read-only skill), so the prompt is not applicable. To skip it: hermes plugins enable model-leaderboard --no-allow-tool-override.

First run seeds the tracked-model list from the main model declared by each installed Hermes profile (default first), then persists it to dashboard/data/models.json. Delete that file to re-seed.

Features

• 📊 Model comparison — context window, input/output/cache prices, Artificial Analysis indices (intelligence, coding, agentic), an SWE-bench Verified column, vision/tools support, open-weights, local VRAM estimate.
• ⭐ Per-profile scoring — one chip per Hermes profile (and default), 0–5 weights on 11 criteria, instant re-ranking + top-5 cards. A profile with no saved criteria starts from a sane default (intelligence 5, price 3, rest 0).
• 📈 Six score views — Cards, Score bars (points earned per criterion), Histograms (grade × weight), Heatmap (grade 0–100, heaviest criteria first; click a cell to enter a real value), Duel (two models, criterion by criterion), and Quality / cost: a scatter with the weighted cost on a log axis, the quality on the other, and the Pareto frontier dashed through the best trade-offs. Each grade says where it came from — OpenRouter, an estimate fitted on the other models, or a value you typed.
• 🤖 Auto-fill — a button in the criteria dialog asks Hermes' own LLM (ctx.llm.complete_structured, the model the user is already running — no subprocess, no API key in the plugin) to propose weights for that profile; the answer is validated against a JSON schema. Never automatic, always user-triggered.
• 🧠 Shipped skill — skill_view('model-leaderboard:refresh-benchmarks') walks an agent through refreshing the hand-curated benchmark indices (SWE-bench Verified, LiveCodeBench) from the public leaderboards and writing them back. Read-only, loaded on demand.
• 🔎 Search — filters the tracked table live and queries OpenRouter for untracked models (checkbox to opt out), with a one-click add.
• 📰 OpenRouter news — side panel: promotions (with end dates), price changes (rolling 1 d / 7 d, configurable threshold), new models (sort by date or score, +1-week pagination).
• 🛡️ Resilience — 5 min OpenRouter cache, stale-on-error fallback on 429s, plus a visibility-gated auto-pull (cheap local poll every 60 s, real refetch every 10 min — only while the page is visible).
• 🧭 Native desktop UX — deletion confirms through the desktop's own ConfirmDialog (imported from the plugin SDK), tooltips and labels fully i18n'd (en/fr), styling left to the app's CSS variables.
• 🔑 No auth plumbing — profiles always persist backend-side; the desktop supplies identity.

Architecture (unified Hermes plugin)

Same conventions as hermes-kanban-gantt:

Reference (docker/llm-dashboard)	This repo
backend/main.py (FastAPI :3004, /api/*)	dashboard/plugin_api.py — FastAPI router mounted at /api/plugins/model-leaderboard/
backend/data/*.json	dashboard/data/ (shipped seed data + runtime state)
frontend/src/App.tsx + components (React 18, Recharts)	src/ (TSX) → esbuild → desktop/plugin.js (artifact committed, SDK/react external, Recharts bundled)

This is the beginning of the README. Read the rest at the original. Read the full README ↗