Mushegh
An open-source framework for building your own 24/7, voice-first, multi-device AI companion — "Mushegh".
Wake-word listening · streaming natural voice · its own reasoning LLM + tools · six-layer memory ·
proactive companion · phone / laptop / glasses · self-improving — self-hosted state, cloud
inference, yours.
📖 Documentation: mushegh.vercel.app — the full docs site
Naming. The project is Mushegh — Modular Universal Synthetic Hyper Engine
General Hub — and the assistant you build and talk to is Mushegh too.
Everything you read is Mushegh: this file, the docs, the website, and the
assistant's own voice. The Python package and CLI deliberately keep the short
internal name mushegh for now (import paths, mushegh-setup, python -m mushegh.…,
the MUSHEGH_ environment prefix and the ~/.mushegh state directory). That is a
separate, planned migration: changing those names without migrating a live
.env would start the brain healthy with zero configuration, which is the one
failure this project's own preflight exists to catch.
NOTICE and LICENSE still carry the older two-name split ("OpenMushegh" the
project, "Mushegh" the persona). That predates the rebrand and does not match the
marks shipped in assets/. Reconciling it is an owner decision, not an edit.
What this is
Mushegh is a framework, not a product. Clone it, run the setup wizard,
point it at the voice/LLM providers you like, and you have a personal assistant you talk to: it
listens for a wake word, answers in a natural streaming voice, remembers across sessions, acts on
your machine and your accounts, reaches you proactively when it matters, and can even improve its
own code — safely and reversibly.
It is built from scratch in Python, runs on a no-GPU laptop, and is designed to be on all the time:
its brain runs as a service on an always-on host (a cheap VPS), while the voice front-end runs
wherever you are — laptop, phone, smart-glasses — all sharing one brain and one memory.
Everything degrades gracefully: a capability with no credentials simply says "that isn't
configured yet" instead of crashing, so you light up integrations one at a time.
Mushegh is its own agent. It has its own reasoning model, its own memory, and its own
personality (personality/Mushegh.md). Pipecat is the voice shell; the LLM + tool loop is the mind.
An optional external multi-agent fleet can be consulted as one tool among many — it never
becomes the assistant.
Where the work is planned
All tracked in this repo. One of them is generated, and says so at the top:
| Document | What it is |
|---|
docs/systems-plan.json | The canonical execution registry: 50 subsystems × floor / raise / elite, every task with a named gate, a verification kind, and the evidence behind its status. Edit this, then render. |
docs/SYSTEMS.md | Generated from the registry — a one-screen dashboard: live counts, per-system status, the next open task. Never edited by hand. |
docs/SYSTEMS-DETAILS.md | The long-form rationale the registry was built from: options weighed with the losers named, budgets, rollbacks. Historical, not authoritative. |
docs/MASTER-PLAN.md | The long-range shape — what this is meant to become. |
TODO.md | The running engineering log: what was built, what it broke, what it measured. It does not move execution state on its own. |
Current position lives in the dashboard's snapshot, generated with it. It is deliberately not
copied here: a hand-maintained score is the first number to go stale.
Why a framework
- Bring your own everything. Cloud quality (ElevenLabs + Deepgram) or 100% local CPU
(Piper/Kokoro + Whisper/Moonshine + openWakeWord) — a flag, not a rewrite. Same for the LLM
(any OpenAI-compatible endpoint: a free proxy, OpenAI, or local Ollama).
- One brain, many devices. The brain is a WebSocket server; a Windows/Linux laptop, a Mac, an
iPhone or Android phone (no app), and Mentra glasses all reach the same agent and memory — over
your Tailnet.
- Safe by construction. Confirm-before-acting is enforced in code (not just prompted),
reversible-only git, repo-scoped self-edits, password-gated privileged routines, a redacted audit
log, deny-by-default fleet. See SECURITY.md.
- Personal but forkable. The character, knowledge, and skills are plain Markdown files you edit;
no code change needed to make it yours.
Table of contents
Feature tour
- Natural voice. Wake on a phrase, understand you across multiple languages (English / French /
German / Russian / Armenian / Ukrainian via Whisper), and reply in a streaming voice with barge-in
(talk over it on headphones).
- Thinks for itself. Its own LLM (any OpenAI-compatible endpoint) with a model fallback chain,
its own personality, and a tool-calling loop — answering directly and fast.
- Remembers. A six-layer memory: the live conversation, durable learned facts, a daily journal,
your Obsidian vault (read and write on the authoritative host), a hot-cache, and optional
semantic recall.
- Acts. Files, processes, PowerShell (incl. elevated), a real visible browser, music, reminders
& phone push, Telegram (read and send), Gmail, Google Calendar, Notion, Home Assistant.
- Looks things up. Weather, crypto, stocks, FX, news, Wikipedia, dictionary, unit/currency
conversion — mostly key-free.
- Reaches you first. A proactive engine that surfaces reminders, calendar events, and alerts
within an interruption budget and quiet hours — speaking if you're listening, pushing if you're not
— and remembers what it said, so you can answer "yes, do that" a minute or ten days later.
- Improves itself. Reads and edits its own source, runs its own test suite, and makes
reversible git commits — with hard guardrails so nothing destructive is possible.
- Controls a PC remotely. An optional edge executor lets the always-on brain drive a laptop
end-to-end (open apps, manage processes, run scripts) from your phone, over your Tailnet.
Architecture
Two cooperating processes over one streaming WebSocket protocol:
┌──────────────── EDGE — laptop / phone / glasses (where you are) ────────────┐
│ Mic → wake word (openWakeWord) → VAD (Silero) → STT (Deepgram/Whisper) │
│ → [BrainBridge] ──WebSocket──┐ │
│ Speaker ← TTS (ElevenLabs/Piper) ←┘ (barge-in, smart turn-taking, AEC) │
└────────────────────────────────────────────┬────────────────────────────────┘
streaming StreamEvents │ (Tailnet + Bearer-token auth)
┌──────────────── BRAIN — always-on host / VPS (24/7) ───────────▼─────────────┐
│ Agent loop: own LLM (+fallback chain) · tool-calling · session + memory │
│ Confirmation tier ENFORCED in code · smart idle session reset │
│ Tools: vault(r/w) · web · telegram · gmail · calendar · notion · smart-home │
│ · utilities · system/PC-control · browser · reminders · coding/git · │
│ Memory L0–L5 · proactive tick (durably logged) · scheduler · audit · health │
│ HTTP sidecar: /talk (iPhone Siri voice) · /control (remote PC executor) │
│ Optionally consults an external multi-agent fleet as ONE tool │
└──────────────────────────────────────────────────────────────────────────────┘
- Edge (
src/mushegh/edge/) keeps audio + STT/TTS local (privacy, low mic latency).
- Brain (
src/mushegh/brain/) holds the 24/7 obligations (reasoning, memory, scheduler,
channels) and can run in-process or as a shared WebSocket server (brain/server.py) so a
laptop, an iPhone, and glasses all share one brain + memory.
- The Pipecat "LLM stage" is replaced by a thin
BrainBridge processor, so swapping reasoning
backends or adopting a different transport never touches the audio pipeline.
🚀 Quick start (5 minutes)
Requires Python 3.11+ and uv.
git clone https://github.com/iamvazghen/Mushegh Mushegh && cd Mushegh
# 1. Install the base + the stack you want (this also makes `mushegh-setup` available):
uv sync --extra edge --extra cloud-voice --extra brain --extra channels --extra identity --extra dev
# ↳ for a 100% local/offline voice stack, drop `cloud-voice` and add `local-voice`.
# 2. Configure interactively — writes your .env:
uv run mushegh-setup
# 3. Verify everything (offline, no keys needed to pass):
uv run python bench/run_all_tests.py # → all green
# 4. Talk to it:
uv run python -m mushegh.edge.assistant # local voice loop on this machine
# …or run the shared brain for phone/glasses:
uv run python -m mushegh.brain.server
uv sync prunes extras you don't list — install the full set you intend to use in one go.
The optional extras are: edge cloud-voice local-voice brain channels browse identity
dev (see pyproject.toml for what each pulls).
After the wizard, SOP.md is the operating manual (setup, running, observability,
subsystems, diagnosis) and TESTING_GUIDE.md is the live end-to-end test
pass: every one-time login/credential, voice enrollment, and the full real-device test plan.
The setup wizard
uv run mushegh-setup (a Rich terminal UI; falls back to plain text on a bare install) walks a forker
through the decisions that matter and writes a ready .env:
- Identity — display name + wake phrase (from the pre-trained set; custom phrases use Porcupine).
- Voice — cloud (ElevenLabs + Deepgram, asks for keys) or local (Piper/Kokoro + Whisper/Moonshine).
- Brain — LLM backend: a free OpenAI-compatible proxy, OpenAI, or local Ollama.
- Knowledge — the path to your Obsidian/Markdown notes (long-term L3 memory).
- Deployment — single machine (loopback) or VPS (binds
0.0.0.0, auto-generates an auth token).
- Security — auto-generates strong passwords for all eight privileged protocols.
- Integrations — optionally wire Telegram, Tavily, Google, Notion, ntfy now (or later).
- Behaviour — proactivity on/off; fleet consult stays off by default.
It never prints a secret back, backs up any existing .env, and preserves every inline comment
in .env.example for the knobs you didn't touch. Re-run it any time to reconfigure.
Networking: the Tailnet requirement
Mushegh is multi-device, and the secure way to connect a phone, a laptop, and glasses to the same
24/7 brain on a VPS is a private mesh VPN — a Tailnet (Tailscale,
WireGuard under the hood). Install Tailscale and sign in on every device — the VPS/brain host,
your laptop, your iPhone, and (via its companion phone) the glasses. Then:
- Every device gets a stable
100.x.y.z address reachable only inside your tailnet.
- The brain binds
0.0.0.0 but is only routable to your own devices — never the public internet —
and is still guarded by the MUSHEGH_API_AUTH_TOKEN bearer the wizard generates.
- The edge connects to
ws://<brain-tailnet-ip>:8765/voice; the iPhone Siri Shortcut posts to
http://<brain-tailnet-ip>:8766/talk; the PC-control executor dials the brain's /control — all
over the tailnet, no port-forwarding, no public exposure.
Rule of thumb: if a device should talk to Mushegh, it must be on the tailnet and logged in.
Off the tailnet, only the public fallbacks work (Telegram bot messages, ntfy push). See the docs
site's Networking page for the exact Tailscale steps.
Devices
One brain, reached many ways — all sharing memory, all over the tailnet:
| Device setup | How it connects | Barge-in | Notes |
|---|
| Laptop (Windows/Linux, built-in mic/speakers) | Mushegh.edge.assistant → brain WS | off (open speakers) | baseline local pipeline |
| Mac (macOS, built-in or external) | Mushegh.edge.assistant → brain WS | off (open speakers) | same Python/PyAudio edge, runs natively |
| Laptop/Mac + headphones (AirPods/BT → host) | same, auto-routes to headphones | on (private) | interrupt mid-sentence |
| iPhone (no app) | "Hey Siri, Mushegh" → POST /talk | n/a | Siri dictation → spoken reply |
| Android (no app) | Assistant/Tasker → POST /talk, or Termux edge-lite | n/a | dictation → spoken reply; or full mic stream via Termux |
| Phone + headphones (AirPods/BT → phone) | same shortcut (android-headphones/phone-headphones hint) | on (private) | reply plays in the earbuds |
| Mentra OS glasses | not shipped — brain-side device routing (device_id="mentra") only | n/a | the TypeScript bridge scaffold was deleted; parked, see TODO.md |
| Home Assistant | brain → HA REST (local) | n/a | states + control (locks confirm-gated) |
| Remote PC control | host executor → brain /control | n/a | brain drives a laptop from anywhere |
Every setup is modelled in src/mushegh/edge/device_profile.py (SUPPORTED_DEVICES: laptop, mac,
iphone, android, airpods, mentra) and verified in bench/test_phase6_multidevice.py. Each has a
step-by-step acceptance test (TTFW numbers, auto-route, barge-in, Siri/Assistant voice, music room,
proactive voice, shared memory) in TESTING_GUIDE.md §14 and the docs
site's Devices page.
Configuration
Everything is driven by environment variables (prefix MUSHEGH_) read from .env. See
.env.example — every knob is documented inline. Nothing is hard-coded; the same
codebase runs CPU-local-only or cloud-quality just by flipping provider flags. Never commit .env
(it's gitignored). The assistant's character is personality/Mushegh.md; its knowledge is the
Markdown under memory/; its skills are skills/*.md — all editable without touching code.
Notable knobs added for safety/memory:
TYPESAFE_API_KEY + MUSHEGH_JEV_ENABLED — optional typed contextual decisions. Jev selects among
explicit options; the generative LLM only phrases the result. Safety, confirmation, privacy and
autonomy budgets remain deterministic code. Without a key, Mushegh asks rather than silently choosing.MUSHEGH_VAULT_WRITABLE — true only on the host that owns the vault; lets Mushegh save notes
into it (create_document with the vault destination). Off on the laptop (its mirror gets clobbered
by the one-way sync).MUSHEGH_SESSION_IDLE_RESET_MINUTES — after this idle gap the brain journals the prior conversation
and clears working memory, so stale context can't bleed into a fresh conversation hours later.
Memory — six layers
| Layer | What | Where |
|---|
| L0 Working | the live conversation (rolling turns; smart-reset on long idle) | RAM |
| L1 Learned | durable facts it saves (remember/recall/forget) | memory/learned/*.md |
| L2 Journal | daily summaries + a durable log of proactive nudges | memory/journal/*.md |
| L3 Vault | your Obsidian knowledge base (read always; write on the authoritative host) | MUSHEGH_VAULT_PATH |
| L4 Hot-cache | fronts the slow paths (search, utilities) | in-process TTL |
| L5 Semantic | recall by meaning, not just keywords | optional local embedder |
L4/L5 are graceful accelerators: the cache needs no service at all; no embedder → keyword recall. The
Markdown layers are always the source of truth.
The tool belt
All tools live in src/mushegh/brain/tools/ (60 modules, 192 tools) and self-degrade when
unconfigured. The whole registry is never advertised at once: 57 core tools are on every turn and
the remaining 33 groups arm only when what you said matches them, under an enforced per-turn token
budget — a turn that would exceed it asks you one short question instead of guessing. Highlights:
- Knowledge —
search_vault, read_vault_note, create_document (save a note on the
authoritative host), web_search (Tavily), scrape_url (Jina Reader — free, keyless),
browse_web (Browserbase).
- Memory —
remember, recall, forget, read_journal.
- Channels —
check_telegram, read_chat, mark_telegram, send_telegram, read_email /
draft_email / send_email (Gmail), notion_search, notion_read_page, notion_append,
notion_comment (pages only). Your task list is Todoist: add_task, list_tasks,
complete_task, update_task, delete_task (the local queue when no token is set).
- Calendar & home —
list_events, create_event, set_home_location; Home Assistant tools.
- Utilities —
weather, news_brief, wiki_lookup,
define_word, convert.
- Media —
play_music (YouTube Music, free; or stream into a Telegram voice chat), stop_music.
- The machine —
file_op, process_op, run_powershell (incl. elevated), browser.
- Proactive —
set_reminder, list_reminders, cancel_reminder, send_push.
- Routines & health —
routine (briefing/focus/lockdown/guest/commute/panic/backup), self_health.
- Self-improvement —
read_source, write_source, run_tests, lint, git_status/diff/log,
git_new_branch/commit/push/revert, list_skills, read_skill.
- The team —
delegate_to_fleet (gated off by default).
Outward-facing / destructive tools are in a confirmation tier that is enforced in code: the agent
holds the call, reads the action back, and runs it only after you say yes. Every tool call is written
to a redacted audit log (audit/*.jsonl).
Proactive companion
A background tick weighs signals (routine, calendar, unread, open threads, self-health) and decides
whether to say something unprompted — within an interruption budget and quiet hours, so
it's helpful, never noisy. It speaks to a listening device or falls back to a Telegram voice note /
ntfy push when you're away. Crucially, every proactive line is recorded — into working memory
(so you can answer "yes, do it" right away) and into the L2 journal (so you can refer back days
later: "that thing you suggested last week"). Mute it on demand by voice: "focus mode for an hour",
"lockdown", "normal", "give me my briefing". Toggle with MUSHEGH_PROACTIVE_ENABLED.
Self-improvement
Mushegh can improve its own codebase, with a hard safety rail: every change is reversible and
verified. Repo-scoped, secret-blocked file I/O; run_tests/lint before trusting a change;
reversible-only git (no reset/force-push/rebase/branch-delete — a revert is a new commit);
writes/commits/pushes are confirm-gated. A background "review" pass also distils durable facts from
conversations into L1 memory every few turns. Ask: "read your self-improvement skill, then make
recall faster." It branches, edits, tests, reads the change back, waits for your yes, commits — and
reverts cleanly if anything regresses.
Testing & benchmarks
uv run python bench/run_all_tests.py — the single gate; the live count is in the dashboard's
snapshot rather than here, so it cannot drift. Each
capability has a hermetic bench/test_*.py (offline, no real network/keys). Network/fleet tests
report SKIP (not FAIL) when their backend is unreachable, so an offline run still passes. The
same suite runs on git push via scripts/githooks/pre-push, so nothing lands red.- Every task in the plan carries a named gate.
bench/test_systems_plan.py fails the build if a
task in docs/systems-plan.json has no gate, if a completed task carries
no evidence, if a planned test is not declared on the task that needs it, or if
docs/SYSTEMS.md has drifted from what the registry renders — a plan that can
drift from the code is a plan that gets believed anyway.
uv run python bench/efficiency_report.py — measures the hot paths (memory recall, cache, brain
TTFT, full-turn latency, prompt size) against efficiency targets. See
SOP.md §9 for how to act on the numbers.
Publish gate: this repo stays private until the suite is green and the efficiency report
meets its targets.
Documentation site
A Next.js documentation site lives in website/ and is built to deploy on Vercel:
cd website
npm install
npm run dev # local preview at http://localhost:3000
# Deploy: import the repo in Vercel, set Root Directory = "website" (framework preset: Next.js).
# or: npm i -g vercel && vercel --cwd website
It documents everything end-to-end: architecture, the setup wizard, the Tailnet requirement, all
device setups (Mentra OS, Home Assistant, laptop, laptop+headphones, phone, phone+headphones),
configuration, memory, security, and licensing.
Deployment
- Run
uv run mushegh-setup, then work through SOP.md (Tailscale on every device,
voice enrollment, VPS ticker, Google/Notion/Telegram logins, GitHub repo, optional
embedder, proactive switch-on) and TESTING_GUIDE.md for the
real-device test plan.
uv run python bench/run_all_tests.py → all green; efficiency_report.py → targets met.- Run the brain as a service on an always-on host and the edge on your laptop (helper scripts
in
scripts/; a systemd unit pattern in deploy/vps/). The VPS ticker delivers recurring
reminders even with the PC off.
Security
Mushegh is powerful — it runs shell commands, drives a browser, sends messages, and edits its own
code. The full safety model (secret handling, enforced confirmation tier, protocol passwords,
self-improvement guardrails, fleet gating, speaker biometrics, audit log, the Tailnet posture, what's
kept out of git) is in SECURITY.md. Read it before deploying, and keep your
fork private if it carries personal memory/ or persona content.
Project layout
src/mushegh/ # package keeps the short internal name `Mushegh`
edge/ # voice pipeline: wake word, VAD, STT/TTS, device routing, barge-in, PC executor
brain/ # the agent: LLM, memory, cache, semantic, proactive, audit, health, protocols
tools/ # the tool belt (one module per capability; SCHEMAS + HANDLERS)
shared/ # the edge↔brain WebSocket protocol
setup_wizard.py # `mushegh-setup` — interactive .env generator
personality/ # Mushegh.md — who Mushegh is (system-prompt persona; edit to make it yours)
memory/ # what it knows (Markdown): about, projects, tools, learned/, journal/
skills/ # on-demand playbooks (self-improvement, architecture, python, pc-control, …)
clients/iphone/ # the phone client assets
website/ # the Next.js documentation site (deploy to Vercel)
deploy/vps/ # the always-on recurring-reminder ticker
scripts/ # edge service install/uninstall helpers
bench/ # the test suite + benchmarks + one-time login helpers
docs/ # systems-plan.json (the plan) + SYSTEMS.md (generated) + SYSTEMS-DETAILS.md,
# MASTER-PLAN, NEXT-UPGRADE-DECISION, home-assistant
TODO.md # the running engineering log — what was done, and what it cost
LICENSE · THIRD_PARTY_NOTICES.md · SECURITY.md · SOP.md · TESTING_GUIDE.md · .env.example
License
Mushegh is released under the MIT License — free to use, modify, and redistribute,
including commercially. You don't "acquire" or pay for MIT; you just keep the LICENSE file (with its
copyright line) in copies of the code.
Dependencies keep their own licenses. Almost all are permissive (MIT / BSD / Apache-2.0 /
Unlicense). The one copyleft dependency is py-tgcalls/ntgcalls (LGPL-3.0), pulled in only
by the optional channels extra for streaming music into a Telegram voice chat — using it as an
unmodified pip library is compatible with shipping your own MIT code, and you can omit it for a
100%-permissive stack. The full breakdown plus cloud-service Terms is in
THIRD_PARTY_NOTICES.md.
The name OpenMushegh is the project's own. Mushegh — the assistant persona — is
referenced only as the blueprint/inspiration and is not used as this project's brand.
Before a public release, run uvx pip-licenses --format=markdown over your locked environment
as a final check.
Project & governance files (all in the repo root):
Contributing
Issues and PRs welcome — see CONTRIBUTING.md for dev setup, the test gate, and
how to add a tool. By participating you agree to the Code of Conduct and the
Acceptable Use policy. Before opening a PR: run
uv run python bench/run_all_tests.py (green) and uv run ruff check src; add a hermetic
bench/test_* check for any new capability. Security issues should be reported privately per
SECURITY.md §10, not as public issues.
