Nexus: firmware for the Waveshare ESP32-S3 audio board
C++17 firmware that turns the Waveshare ESP32-S3 audio board into a
voice-controlled music box: say the wake word and talk to Google's Gemini
Live, play music from YouTube (via Invidious) with songs saved to the SD
card, set alarms, timers and reminders, record audio, and control it all
from a web dashboard the device serves itself at http://nexus.local.
Features
- Voice assistant — on-device wake word (ESP-SR, with echo
cancellation) opens a live, two-way audio session with Gemini Live over a
direct WebSocket. Gemini can call 25 device tools: play music, control
playback and volume, set alarms, timers and reminders, set the LEDs, keep
notes.
- Music — search and stream YouTube audio through Invidious, with a
queue, autoplay of related songs, and exact seeking. With caching on,
songs are saved to the SD card while they play and replay from the card.
- Alarms, timers, reminders — an alarm takes over the speaker (pausing
music and ending an assistant session), plays a library song or a built-in
tone, snoozes, and restores the music afterwards.
- Recorder — records both mics (stereo) or the cleaned-up voice signal
as Ogg Opus; recordings can be played in the browser or on the device,
renamed and downloaded.
- Web dashboard — player, library, queue, assistant, alarms,
recordings, files and settings, served from flash with over-the-air
updates for both the dashboard and the firmware
(waveshare-dashboard).
- On-device screen (optional) — LVGL touchscreen UI.
- Robust storage — settings, state, alarms and the recordings list live
in small append-only databases on the card that survive power cuts.
Hardware
| Part | |
|---|
| MCU | ESP32-S3, dual-core 240 MHz, Wi-Fi, 16 MB flash, 8 MB octal PSRAM |
| Audio | ES7210 mic ADC (two MEMS mics + echo reference), ES8311 codec, class-D speaker amplifier |
| Storage | microSD (SDMMC, FAT32) |
| Controls | Five keys through a TCA95xx I2C expander, WS2812 LED ring |
| Display (optional) | 320×240 ILI9341 SPI screen with XPT2046 touch (CONFIG_DISPLAY_ENABLE) |
Getting started
- Toolchain: ESP-IDF v6.0.1.
. $IDF_PATH/export.sh
idf.py build
idf.py -p /dev/ttyACM0 flash monitor
idf.py menuconfig → Waveshare Audio Development Board Config holds the
Wi-Fi, wake word, SD card and display options. PSRAM and the partition
table must be enabled as in sdkconfig.defaults plus your board's needs.
- SD card: a FAT32 microSD card in the slot.
- Credentials (stored in NVS, never on the card), any of:
- put
wifi_config.json ({"ssid": "...", "password": "..."}) and
gemini_config.json ({"api_key": "..."}) on the card and boot: they
are imported and removed from the card;
- with no Wi-Fi configured (or after five failed connection attempts),
the device opens a setup access point: join it and use the portal;
POST /api/config/gemini and POST /api/wifi/configure from the
dashboard's settings.
- Dashboard: open
http://nexus.local (or the device's IP). If no
dashboard bundle is installed yet, a built-in page is served at / and
always at /recovery. Build and upload the dashboard from its repository
with npm run deploy.
- Updates: later firmware goes over the air:
curl -X POST --data-binary @build/waveshare.bin http://nexus.local/api/ota.
How it fits together
keys / wake word / dashboard (/api/ws, REST) / Gemini tool calls
│
AppController, HTTP routes
│
┌──────────────┬──────┴───────┬───────────────┬──────────────┐
Assistant Music Alarms Recorder Storage
(gemini_live) (media_player) (services/ (app/audio/ (nexus_db,
alarm) recording) sd_storage)
└──────────────┴──────┬───────┴───────────────┘
AudioOrchestrator (who owns the speaker)
│
audio_core: mic capture, AFE + wake word, mixer, I2S
All live state sits in one in-RAM store, EmbeddedSysDb: services change
it through mutate(), and tasks that care (the dashboard channel, LEDs,
audio, persistence) are woken by per-field change bits. What must survive a
restart is saved to system.ndb on the card.
| Part | README |
|---|
| Audio I/O, wake word, mixer, orchestrator | components/audio_core |
| System state, ring buffers, task layout | components/core_sysdb |
| Music player, seeking, library | components/media_player |
| Voice assistant and tools | components/gemini_live |
| Databases on the card | components/nexus_db |
| SD card access | components/sd_storage |
| Credentials | components/credentials |
| HTTP server core | components/http_server |
| Screen UI | components/ui_view |
| App logic, keys, recorder | main/app |
| Alarms, timers, reminders | main/services/alarm |
Documentation
Related repositories
License
MIT, see LICENSE.
