OVOS Plugin Arena

Which plugin should I use? This arena answers with data.

Live site: openvoiceos.github.io/ovos-plugin-arena

See every page in the visual site tour.

This is a GitHub-native benchmarking and voting arena for OpenVoiceOS plugins. Reproducible benchmark scripts rank plugins on labelled datasets. Blind A/B battles let humans refine those rankings. No servers, no databases, no accounts. The repository is the arena:

• every fighter (competitor) is a JSON file in registry/competitors/
• every dataset is a JSON file in registry/datasets/
• every benchmark is one reproducible Python script in benchmarks/
• benchmark predictions are published to HuggingFace datasets
• votes are GitHub issues, tallied by a scheduled Action
• leaderboards are JSON files committed by CI and served by GitHub Pages

Status

[!WARNING] Work in progress. Development is focused on the platform itself. The tasks, datasets and fighter configurations are still being defined, so every leaderboard, rating and vote on the site and in the prediction datasets is placeholder data. At launch the results will be regenerated and the votes reset.

Alpha: all modalities wired. Every league has a reproducible benchmark script and registry fighters:

League	Benchmark	Ranking signal
intent_zero_shot · intent_online · intent_offline · intent_keyword	benchmarks/intent_*.py over intents-for-eval (12 langs) + massive-templates (52 langs)	accuracy / macro-F1 / OOD-FPR / slot-EM → ELO seed
stt	benchmarks/stt_minds14.py over MInDS-14	WER → ELO seed
wake_word	benchmarks/ww_hey_mycroft.py over synthetic-wakewords	detection error / false-accept / false-reject → ELO seed
tts	benchmarks/tts_intents_prompts.py	mean UTMOS, with STT round-trip intelligibility WER beside it → ELO seed

The intent leagues are fully populated with published predictions. STT, TTS and wake-word fighters and datasets are registered and runnable, and prediction sweeps run on an ongoing basis: boards render as swept rows land. Pages deployment is live at https://openvoiceos.github.io/ovos-plugin-arena/.

Transparency

• AI usage: this project is developed with AI coding agents (Claude) under human direction and review. Benchmark numbers are not AI-generated: every prediction row comes from actually running the real OVOS plugins over the published datasets via the scripts in benchmarks/, and every run is reproducible from a pinned dataset revision recorded in each row.
• Votes: the vote log is the public GitHub issue history. It is auditable and deterministically replayable at any time.

How it works

registry/*.json ──┐
                  ▼
benchmarks/<bench>.py        one script per benchmark: trains each fighter,
   │                         predicts the test split, publishes JSONL rows
   ▼
HF repos (1 per modality)    predictions/<lang>/<competitor>.jsonl, split per lang
   │
   ▼  assemble.yml (daily)
frontend-static/public/data/ battles-*.json     blind A/B pools
                             benchmark-*.json   auto-metric boards
                             elo-seed-*.json    benchmark-derived initial ELO
                             leaderboard-*.json ELO boards
                             competitors.json   fighter bestiary
   │
   ▼  Astro build → GitHub Pages
voter picks A/B  →  prefilled GitHub issue  →  tally.yml (hourly)
                                                parses, dedupes, replays ELO,
                                                commits boards, closes issues

ELO seeding: before any human vote exists, the ELO board is seeded by deterministic auto-battles derived from the benchmark metrics (one auto-battle per sample where exactly one fighter is correct, at ¼ K-factor). Human votes then move ratings at full weight on top of the seed.

Determinism (§P5): battle ids are content hashes, so re-running assemble never invalidates open votes. Both the seed and the vote replay are fully deterministic, so the standings are reproducible from public data alone.

The fighters

Each fighter is a shippable configuration. Its config is a valid mycroft.conf fragment: an intents section with a tier-suffixed pipeline plus per-plugin config blocks. Single-stage pipelines benchmark one engine in the league its training regime implies. Multi-stage pipelines are fusion fighters, filed by their heaviest stage, under portmanteau names: Padapt (Padatious × Adapt, the stock OVOS cascade) and Nebulapt (Nebulento × Adapt). Fighters carry a species (the plugin class they instantiate), architecture types (GOFAI, fuzzy-match, neural-net, ensemble, and others), a size class (micro to titan), and a procedurally generated sprite derived from their id hash. Browse the bestiary on the Fighters page or in registry/competitors/. All fighters run end-to-end through the real OPM pipeline plugins. The plugin owns its confidence thresholds; the arena owns none.

Running a benchmark

pip install ".[hf,audio]"   # audio extra (numpy/soundfile/pyarrow) for stt/ww/tts
# intent (shared engine in runner/intent_bench.py)
python benchmarks/intent_intents_for_eval.py                  # full run (CPU, ~15 min)
python benchmarks/intent_intents_for_eval.py \
    --competitors padatious-medium --langs en-US --max-samples 50   # smoke run
# audio modalities (shared engine in runner/media_bench.py)
python benchmarks/stt_minds14.py --dataset minds14-en-US --max-samples 50
python benchmarks/ww_hey_mycroft.py --competitors openwakeword-hey-mycroft
python benchmarks/tts_intents_prompts.py --langs en-US --max-samples 30
python benchmarks/stt_minds14.py --upload                    # publish to HF

Every benchmark, intent and audio, shares the same flags (--competitors, --langs, --max-samples, --dataset, --upload). Runs are resumable, and rows carry the pinned dataset revision, plugin version, and runner version. Audio benchmarks instantiate the real OVOS STT/TTS/wake-word plugins offline. Uploading requires HF write access to the results repo. See docs/benchmarks.md for per-modality details.

Assembling the arena locally

pip install ".[hf]"
python -m arena.cli assemble --predictions OpenVoiceOS/ovos-intent-bench-intents-for-eval
python -m arena.cli export-bestiary
python -m arena.cli export-index
python -m arena.cli tally --keep-issues-open   # dry-run vote tally

cd frontend-static
npm install
npm run dev      # http://localhost:4321/ovos-plugin-arena

assemble also accepts a local predictions directory instead of an HF repo id.

Tests

pip install ".[test]"
python -m pytest tests/ -q

Fork your own arena

1. Fork this repo. Enable Actions, and set Settings → Pages → Source to "GitHub Actions" (public repo required for Pages).
2. Set the HF_PREDICTIONS repo variable to your prediction dataset(s) (comma-separated), and ASTRO_SITE / ASTRO_BASE to your Pages URL.
3. Drop your fighters and datasets as JSON files in registry/, add a benchmark script in benchmarks/, and publish predictions to HF.
4. Run the Assemble battles workflow once. Voters take it from there.

Documentation

Full docs live in docs/. Start there for the specification, the local-testing guide, and the fighter walkthrough.

Key files

Path	Purpose
registry/	Declarative fighters + datasets (JSON) and their loaders
benchmarks/	One reproducible prediction script per benchmark
runner/	Plugin adapters + shared bench engines: intent_bench (intent), media_bench + stt_bench/ww_bench/tts_bench (audio)
arena/	Core library: prediction loading, metrics, battles, ELO, CLI
frontend-static/	Astro static site (leaderboards, battles, bestiary)
.github/workflows/assemble.yml	Daily data refresh from HF predictions
.github/workflows/tally.yml	Hourly vote tally → leaderboards
.github/workflows/pages.yml	Astro build + Pages deploy
.github/ISSUE_TEMPLATE/vote.yml	The vote issue form (applies the vote label)
docs/SPECIFICATION.md	Full specification
docs/add-a-fighter.md	How to add your plugin as a fighter
docs/operations.md	Maintainer runbook for the vote loop (verify, audit, replay, troubleshoot)
docs/reproduce-a-row.md	Reproduce one leaderboard row end to end in about five minutes

---

Credits

Developed by TigreGótico for OpenVoiceOS.

This project was funded through the NGI0 Commons Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 101135429.