OmniAgent

Your AI representative across communication channels.

OmniAgent is a personal AI assistant that routes messages across multiple communication platforms, processes them via an AI agent, and responds on your behalf.

Features

• 💬 Multi-Channel Support - Telegram, Discord, Slack, WhatsApp, and more
• 🤖 AI-Powered Responses - Powered by omnillm (Claude, GPT, Gemini, etc.)
• 🎤 Voice Notes - Transcribe incoming voice, respond with synthesized speech via OmniVoice
• 📞 Full-Duplex Phone Calls - Real-time phone conversations via Twilio Media Streams
• ⚡ Native Voice-to-Voice - Ultra-low latency (~100-300ms) via OpenAI Realtime, Gemini Live, or Deepgram Agent APIs
• 🧩 Skills System - Markdown skills (OpenClaw compatible) and compiled Go skills
• 💾 Persistent Sessions - Conversation history with SQLite storage via omnistorage-core
• 🧠 Semantic Memory - Multi-backend memory with automatic recall via omnimemory
• ⏰ Scheduled Jobs - Cron expressions, intervals, and one-time job scheduling
• 🔒 Secure Sandboxing - WASM and Docker isolation with GPU passthrough
• 🌐 Browser Automation - Built-in browser control with dialog handling via Rod
• 🔌 WebSocket Gateway - Real-time control plane with tools RPC endpoint
• 📊 Observability - Integrated tracing via omniobserve
• 🎭 Agent Profiles - Bootstrap profiles and lean mode for resource optimization
• 🛡️ Access Policies - Per-sender tool access control and channel conformance
• 🔐 Vault Credentials - Secure credential storage via 1Password, Bitwarden, file, or environment
• 🔑 Skill Secrets - GitHub-style requires.secrets declaration in SKILL.md, global/per-skill config bindings, per-skill-type injection (MCP, OpenAPI, compiled), required-secret gating, and log/output redaction
• 🔗 OpenAI-Compatible API - Drop-in replacement for OpenAI client libraries with SSE streaming
• 🖼️ Image Generation - AI image generation via OpenAI (DALL-E) or Fal AI (FLUX)
• 👥 Multi-Agent Support - Run multiple agents with different models and configurations
• 🏢 Team Mode - Multi-user deployments with magic-link, email+password, or Google/GitHub sign-in, an admin UI, private/group chats, and PostgreSQL row-level isolation
• 🪪 Virtual Agents - Named personas with per-agent skills, agent-scoped secrets, owner/maintainer roles, and a discoverable catalog
• 🛒 Agent Marketplace Primitives - Reusable agent and skill listings, filters, and provider interfaces for host apps such as UIForge
• 📈 Usage Analytics - Token usage tracking, tool call statistics, and cost estimation
• 🔧 Tool Visualization - Real-time tool call display with arguments and results in web UI

Ways to Connect

OmniAgent supports multiple connection methods for different use cases:

Method	Protocol	Use Case	Latency
WhatsApp	WebSocket	Personal messaging, voice notes	Text: instant, Voice: ~2s
Telegram/Discord/Slack	Bot API	Team messaging, notifications	Instant
OpenAI-Compatible API	HTTP/SSE	Programmatic access, web apps	Streaming
Phone Calls	Twilio/Telnyx	PSTN voice conversations	Traditional: 500ms+, Realtime: ~100ms
LiveKit (WebRTC)	WebRTC	Browser/mobile voice, meetings	~100-300ms
Meetings by Phone	Twilio/Telnyx	Attend Zoom, Meet, and Teams meetings by dial-in; transcript and summary	Live transcript

Quick links:

• WhatsApp Setup - Scan QR code to connect
• OpenAI API - Use any OpenAI client library
• Voice Gateway - Phone calls via Twilio/Telnyx
• LiveKit Agents - WebRTC voice in meetings
• Meetings by Phone - omniagent meeting join --invite meeting.ics attends a meeting and writes notes

Screenshots

Stock Skill via OmniAgent Web Interface

Search Skill via WhatsApp

Installation

go install github.com/plexusone/omniagent/cmd/omniagent@latest

Quick Start

WhatsApp + OpenAI

The fastest way to get started is with WhatsApp and OpenAI:

# Set your OpenAI API key
export OPENAI_API_KEY="sk-..."

# Run with WhatsApp enabled
OMNIAGENT_AGENT_PROVIDER=openai \
OMNIAGENT_AGENT_MODEL=gpt-4o \
WHATSAPP_ENABLED=true \
omniagent gateway run

A QR code will appear in your terminal. Scan it with WhatsApp (Settings -> Linked Devices -> Link a Device) to connect.

Configuration File

For more control, create a configuration file:

# omniagent.yaml
gateway:
  address: "127.0.0.1:18789"

agent:
  provider: openai          # or: anthropic, gemini
  model: gpt-4o             # or: claude-sonnet-5, gemini-2.0-flash
  api_key: ${OPENAI_API_KEY}
  system_prompt: "You are OmniAgent, responding on behalf of the user."

channels:
  whatsapp:
    enabled: true
    db_path: "whatsapp.db"  # Session storage

  telegram:
    enabled: false
    token: ${TELEGRAM_BOT_TOKEN}

  discord:
    enabled: false
    token: ${DISCORD_BOT_TOKEN}

  twilio_sms:
    enabled: false
    account_sid: ${TWILIO_ACCOUNT_SID}
    auth_token: ${TWILIO_AUTH_TOKEN}
    phone_number: ${TWILIO_PHONE_NUMBER}
    messaging_service_sid: ${TWILIO_MESSAGING_SERVICE_SID}  # Optional: enables RCS with SMS/MMS fallback
    webhook_path: /webhook/twilio/sms  # Supports SMS, MMS, and RCS

voice:
  enabled: true
  response_mode: auto        # auto, always, never

  # Option 1: Native voice-to-voice (lowest latency, ~100-300ms)
  realtime:
    provider: openai         # or: gemini, deepgram
    voice: alloy             # OpenAI: alloy, nova, etc. Gemini: Puck, Charon, etc. Deepgram: aura-2-thalia-en, etc.

  # Option 2: Traditional pipeline (custom STT/TTS providers)
  # stt:
  #   provider: deepgram
  #   model: nova-2
  # tts:
  #   provider: elevenlabs
  #   voice_id: your-voice-id

skills:
  enabled: true
  paths:                     # Additional skill directories
    - ~/.omniagent/skills
  max_injected: 20           # Max skills to inject into prompt

Run with the config file:

omniagent gateway run --config omniagent.yaml

OpenAI-Compatible API

OmniAgent exposes an OpenAI-compatible REST API, allowing you to use standard OpenAI client libraries:

# Start the gateway with API enabled
omniagent gateway run --config omniagent.yaml

# Use with any OpenAI client
curl http://localhost:18789/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "model": "omniagent",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

API Endpoints

Endpoint	Description
POST /openai/v1/chat/completions	Chat completions with SSE streaming
GET /openai/v1/models	List available models
POST /openai/v1/images/generations	Generate images from prompt
POST /openai/v1/images/edits	Edit images with mask
POST /openai/v1/images/variations	Create image variations
GET /api/v1/tools	List registered tools
GET /api/v1/agents	List configured agents
POST /api/v1/agents	Create a new agent
GET /api/v1/cron/jobs	List scheduled jobs
GET /api/health	Health check
GET /docs	Interactive API documentation (Scalar)
GET /api/openapi.json	OpenAPI 3.1 specification

Python Example

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:18789/openai/v1",
    api_key="your-api-key"
)

response = client.chat.completions.create(
    model="omniagent",
    messages=[{"role": "user", "content": "What tools do you have?"}],
    stream=True
)

for chunk in response:
    print(chunk.choices[0].delta.content, end="")

See OpenAI API Guide for detailed documentation.

Skills

OmniAgent supports skills compatible with the OpenClaw SKILL.md format. Skills extend the agent's capabilities by injecting domain-specific instructions into the system prompt.

Managing Skills

# List all discovered skills
omniagent skills list

# Show details for a specific skill
omniagent skills info sonoscli

# Check requirements for all skills
omniagent skills check

Skill Format

Skills are defined in SKILL.md files with YAML frontmatter:

---
name: weather
description: Get weather forecasts
metadata:
  emoji: "🌤️"
  requires:
    bins: ["curl"]
    secrets:
      - name: WEATHER_API_KEY
        description: API key for the weather provider
        required: true
  install:
    - name: curl
      brew: curl
      apt: curl
---

# Weather Skill

You can check the weather using the `curl` command...

Skill Discovery

Skills are discovered from:

1. Built-in skills directory
2. ~/.omniagent/skills/
3. Custom paths via skills.paths config

Skills with missing requirements (binaries, env vars) are automatically skipped. A skill that declares a required secret (requires.secrets) with no matching binding — see Secrets below — is skipped the same way, with a logged reason instead of failing later at call time.

Compiled Skills

For better performance and type safety, register Go functions as LLM tools:

import (
    "github.com/plexusone/omniagent/agent"
    "github.com/plexusone/omniagent/skills/compiled"
)

// Create a skill with tools
type WeatherSkill struct{}

func (s *WeatherSkill) Name() string        { return "weather" }
func (s *WeatherSkill) Description() string { return "Weather forecasts" }
func (s *WeatherSkill) Tools() []compiled.Tool {
    return []compiled.Tool{{
        Name:        "get_weather",
        Description: "Get weather for a location",
        Parameters: map[string]compiled.Parameter{
            "location": {Type: "string", Required: true},
        },
        Handler: func(ctx context.Context, params map[string]any) (any, error) {
            return fetchWeather(params["location"].(string))
        },
    }}
}
func (s *WeatherSkill) Init(ctx context.Context) error { return nil }
func (s *WeatherSkill) Close() error                   { return nil }

// Register with agent
agent.New(config, agent.WithCompiledSkill(&WeatherSkill{}))

Remote Skills

Remote skills connect to external services and expose their capabilities as agent tools.

MCP Skills - Spawn external MCP servers and expose their tools:

import "github.com/plexusone/omniagent/skills/remote/mcp"

agent, err := agent.New(config,
    agent.WithMCPSkill(mcp.Config{
        Name:    "github",
        Command: []string{"npx", "-y", "@modelcontextprotocol/server-github"},
        Env: map[string]string{
            "GITHUB_TOKEN": os.Getenv("GITHUB_TOKEN"),
        },
    }),
)

OpenAPI Skills - Parse OpenAPI 3.x specs and expose operations as tools:

import openapi "github.com/plexusone/omniagent/skills/remote/openapi"

agent, err := agent.New(config,
    agent.WithOpenAPISkill(openapi.Config{
        Name:    "petstore",
        SpecURL: "https://petstore3.swagger.io/api/v3/openapi.json",
        Auth: openapi.AuthConfig{
            Type:   openapi.AuthBearer,
            Token:  os.Getenv("API_TOKEN"),
        },
    }),
)

See the Skills Guide for configuration options.

Skill Secrets

A skill declares the secrets it needs in SKILL.md frontmatter (requires.secrets, shown above); OmniAgent supplies them, GitHub-Actions style. Values come from global or per-skill config bindings, resolved the same way as any other credential (plain values or op:///bw:/// file:///env:// vault URIs):

# omniagent.yaml
secrets:
  GITHUB_TOKEN: "op://Shared/github/token"   # global binding

skills:
  config:
    github:
      secrets:
        GITHUB_TOKEN: "env://GITHUB_TOKEN_OVERRIDE"  # wins over the global binding

Resolved values are injected into MCP subprocess environments, OpenAPI auth, and compiled skills via agent.WithSecretEnv; a skill with an unmet required secret is excluded from the loaded set (with a logged reason) instead of loading and failing later. Every resolved value — vault-backed or plain — is masked out of log output, including omniagent config show. In team mode, secrets are instead managed per-agent from the web UI (see Team Mode below); superadmins get a read-only view of the global config bindings under Admin. Full reference: Secrets.

Roles

Roles are high-level agent personas that combine skills, workflows, and system prompts into cohesive behaviors. They separate organizational responsibilities from runtime implementations.

import (
    "github.com/plexusone/omniagent/agent"
    "github.com/plexusone/omniagent/agent/roles"
    facilitator "github.com/plexusone/omnirole-facilitator"
)

// Create a role with configuration
pmRole := facilitator.New(facilitator.Config{
    DefaultConfluenceSpace: "TEAM",
    EnableActionTracking:   true,
})

// Create role manager with skills
mgr, _ := roles.NewManager(pmRole, meetingSkill, googleSkill, confluenceSkill)
mgr.Init(ctx)
defer mgr.Close()

// Access role capabilities
prompt, _ := mgr.SystemPrompt(ctx)
workflows := mgr.Workflows()
spec := mgr.Spec()

// Use policy enforcement
if err := mgr.CheckToolAccess(ctx, "confluence_publish"); err != nil {
    // Tool access denied by policy
}

// Context-aware behaviors
mgr.SetBehaviorContext(role.BehaviorContextMeeting)
behaviors := mgr.GetActiveBehaviors(ctx)

// Track metrics
mgr.RecordMetric(ctx, "meetings-facilitated", 1)

Available Roles

Role	Package	Description
Meeting PM	github.com/plexusone/omnirole-facilitator	Meeting facilitation, notes, action tracking

Role Features

Feature	Description
Behaviors	Context-aware actions (meeting, chat, autonomous)
Policies	Tool access control, data access, rate limits
Metrics	KPIs and success measurements
Delegation	Sub-agent orchestration
Workflows	Structured multi-step operations

See the Roles Guide for complete documentation.

Team Mode

Team mode turns a single-operator deployment into a multi-user one: real accounts, allowlist-closed sign-in (magic-link email, email+password, or Google/GitHub), private and group chats, and virtual agents that people discover in a catalog and chat with.

# omniagent.yaml
team:
  enabled: true
  superadmin_email: [email protected]
  base_url: https://team.example.com
  database:
    app_dsn: postgres://omniagent_app:pw@db:5432/omniagent_team
    migrate_dsn: postgres://owner:pw@db:5432/omniagent_team
  smtp:
    host: smtp.example.com
    port: 587
    from: [email protected]
web:
  enabled: true
agent:
  provider: anthropic
  model: claude-sonnet-5
  # api_key: set via ANTHROPIC_API_KEY (loadEnv falls back to the
  # provider-specific env var when api_key is left unset here)

omniagent gateway run --config omniagent.yaml

The gateway migrates the database (schema and PostgreSQL row-level security), serves the SPA at /, and emails magic links (or logs them in dev when no SMTP is set). The account in superadmin_email bootstraps as superadmin on first sign-in, and gets an Admin tab to manage the allowlist and members.

Virtual agents. In team mode an agent is a first-class entity: a persona + a chosen subset of the deployment's skills + agent-scoped secrets, with per-agent owner/maintainer roles (independent of chat membership — conversing never grants configuration) and a private/listed + featured registry. Owners configure agents under My Agents, anyone browses the Catalog and starts a DM or group, and superadmins promote agents under Curation.

• Uses PostgreSQL for production (row-level security isolation); a non-postgres:// app_dsn selects SQLite for local trials only.
• Agent-scoped secrets (team.secrets) are namespaced per agent so two agents load disjoint secrets with no cross-leak; superadmins get a read-only view of the deployment-wide config bindings (names and set-state, never values) under Admin.
• Email+password sign-in (argon2id-hashed) is available alongside magic-link, seeded by an optional team.superadmin_password bootstrap.
• Optional Google OIDC and GitHub OAuth sign-in (team.sso.*), additive to magic-link email — an SSO identity links by verified email to an existing account rather than creating a duplicate.
• A production Docker Compose stack (Caddy + omniagent + PostgreSQL) for a single-VM deployment lives under deploy/team/prod/.

See the Team Mode guide, Virtual Agents guide, and Agent Marketplace guide, Team Deployment guide for the full walkthrough.

Sessions

OmniAgent supports persistent conversation sessions. omniagent gateway run wires this up from config — no code required:

storage:
  type: sqlite # or redis, memory
  path: /data/omniagent.db
sessions:
  enabled: true

Embedding binaries that build their own *agent.Agent use the library API:

import (
    "github.com/plexusone/omniagent/agent"
    "github.com/plexusone/omnistorage-core/kvs/backend/sqlite"
)

// Create storage backend
backend, _ := sqlite.New(sqlite.Config{Path: "omniagent.db"})

// Create agent with sessions
a, _ := agent.New(config,
    agent.WithSessionsFromStorage(backend),
)

// Process with conversation history
response1, _ := a.ProcessWithSession(ctx, "user-123", "My name is Alice")
response2, _ := a.ProcessWithSession(ctx, "user-123", "What's my name?")
// Agent remembers: "Your name is Alice"

See Sessions Guide for details.

Scheduled Jobs

OmniAgent supports scheduled job execution via the cron package. omniagent gateway run starts the scheduler automatically in single-agent mode, using the same storage: config as Sessions above. The library API, for embedding binaries:

import (
    "github.com/plexusone/omniagent/agent"
    "github.com/plexusone/omnistorage-core/kvs/backend/sqlite"
)

// Create agent with cron support
backend, _ := sqlite.New(sqlite.Config{Path: "omniagent.db"})
a, _ := agent.New(config,
    agent.WithSessionsFromStorage(backend),
    agent.WithCronScheduler(),
)

The LLM can then create scheduled jobs via tool calls:

Tool	Description
cron_create	Create a new scheduled job
cron_list	List all jobs (filterable by status)
cron_get	Get job details
cron_delete	Delete a job
cron_enable	Enable a disabled job
cron_disable	Disable without deleting
cron_trigger	Run job immediately

Schedule types:

• Cron expressions: 0 0 9 * * * (9am daily, with seconds)
• Intervals: 1h, 30m, 24h
• One-time: RFC3339 timestamp for single execution

Action types:

• send_message - Send a message to a session
• call_webhook - Make an HTTP request
• call_tool - Invoke a registered tool

See Cron Guide for details.

Sandboxing

OmniAgent provides layered security for tool execution:

App-Level Permissions

Capability-based permissions control what tools can do:

• fs_read - Read files from allowed paths
• fs_write - Write files to allowed paths
• net_http - Make HTTP requests to allowed hosts
• exec_run - Execute allowed commands

Docker Isolation

For OS-level isolation, tools can run inside Docker containers:

sandbox, _ := sandbox.NewDockerSandbox(ctx, sandbox.DockerConfig{
    Image:       "alpine:latest",
    NetworkMode: "none",           // No network access
    CapDrop:     []string{"ALL"},  // Drop all capabilities
    Mounts: []sandbox.DockerMount{
        {HostPath: "/tmp/data", ContainerPath: "/data", ReadOnly: true},
    },
}, &appConfig)

result, _ := sandbox.Run(ctx, "cat", []string{"/data/file.txt"})

GPU Passthrough

For GPU-accelerated workloads, enable NVIDIA GPU passthrough:

sandbox, _ := sandbox.NewDockerSandbox(ctx, sandbox.DockerConfig{
    Image: "nvidia/cuda:12.0-base",
    GPU: &sandbox.GPUConfig{
        Enabled:      true,
        DeviceIDs:    []string{"0"},
        Capabilities: []string{"compute", "utility"},
    },
})

WASM Runtime

For lightweight isolation, tools can run in a WASM sandbox (wazero):

runtime, _ := sandbox.NewRuntime(ctx, sandbox.Config{
    Capabilities:  []sandbox.Capability{sandbox.CapFSRead},
    MemoryLimitMB: 16,
    Timeout:       30 * time.Second,
    AllowedPaths:  []string{"/tmp/data"},
})

Agent Profiles

Profiles customize agent behavior for different use cases:

import "github.com/plexusone/omniagent/agent/profiles"

profile := &profiles.BootstrapProfile{
    Name:               "customer-support",
    SystemPromptPrefix: "You are a customer support agent.\n",
    AllowedTools:       []string{"search_kb", "create_ticket"},
    DeniedTools:        []string{"shell", "browser"},
}

a, _ := agent.New(config, agent.WithProfile(profile))

Lean Mode

Optimize for constrained environments:

leanMode := profiles.NewLeanMode(profiles.LeanLevelModerate)
a, _ := agent.New(config, agent.WithLeanMode(leanMode))

Level	Memory Reduction	Use Case
Off	None	Default operation
Light	~15%	Slightly constrained
Moderate	~35%	Mobile/embedded
Aggressive	~60%	Severely constrained

See Agent Profiles Guide for details.

Access Policies

Tool Policies

Control which tools are available per sender:

import "github.com/plexusone/omniagent/tools/policy"

manager := policy.NewManager()
manager.SetPolicy("guest", &policy.Policy{
    AllowedTools: []string{"search", "weather"},
    DeniedTools:  []string{"shell", "browser"},
    RateLimit: &policy.RateLimit{
        MaxCalls: 10,
        Window:   time.Minute,
    },
})

Channel Policies

Validate messages against content rules:

import "github.com/plexusone/omniagent/channels/policy"

checker := policy.NewConformanceChecker(config)
checker.AddRule(policy.ConformanceRule{
    Name:    "rate-limit",
    Action:  policy.ActionRateLimit,
    RateLimit: &policy.RateLimit{MaxMessages: 60, Window: time.Minute},
})

See Access Policies Guide for details.

Environment Variables

Variable	Description
OPENAI_API_KEY	OpenAI API key
ANTHROPIC_API_KEY	Anthropic API key
GEMINI_API_KEY	Google Gemini API key
OPENROUTER_API_KEY	OpenRouter API key
OMNIAGENT_AGENT_PROVIDER	LLM provider: openai, anthropic, gemini, openrouter
OMNIAGENT_AGENT_MODEL	Model name (e.g., gpt-4o, claude-sonnet-5)
WHATSAPP_ENABLED	Set to true to enable WhatsApp
WHATSAPP_DB_PATH	WhatsApp session storage path
TELEGRAM_BOT_TOKEN	Telegram bot token (auto-enables Telegram)
DISCORD_BOT_TOKEN	Discord bot token (auto-enables Discord)
TWILIO_ACCOUNT_SID	Twilio Account SID (auto-enables SMS/MMS/RCS)
TWILIO_AUTH_TOKEN	Twilio Auth Token
TWILIO_PHONE_NUMBER	Twilio phone number in E.164 format
TWILIO_MESSAGING_SERVICE_SID	Messaging Service SID for RCS (enables RCS with SMS/MMS fallback)
TWILIO_WEBHOOK_PATH	SMS webhook path (default: /webhook/twilio/sms)
SERPER_API_KEY	Serper API key for web search
SERPAPI_API_KEY	SerpAPI key for web search (alternative)
DEEPGRAM_API_KEY	Deepgram API key for voice STT/TTS (traditional)
OMNIAGENT_VOICE_ENABLED	Set to true to enable voice processing
OMNIAGENT_VOICE_RESPONSE_MODE	Voice response mode: auto, always, never
OMNIAGENT_VOICE_REALTIME_PROVIDER

Vault-Backed Credentials

OmniAgent supports storing credentials in password managers via omnivault and omnitoken.

Supported Vault Providers

Provider	URI Scheme	Environment Variable
1Password	op://	OP_SERVICE_ACCOUNT_TOKEN
Bitwarden	bw://	BW_ACCESS_TOKEN, BW_ORGANIZATION_ID
File	file://	-
Environment	env://	-

Static Credentials

API keys and tokens can be stored in vaults instead of config files:

# omniagent.yaml
agent:
  provider: anthropic
  model: claude-sonnet-5
  api_key: "op://MyVault/anthropic/api-key"  # Resolved from 1Password

channels:
  telegram:
    enabled: true
    token: "bw://org-id/telegram-bot-token"  # Resolved from Bitwarden

  discord:
    enabled: true
    token: "env://DISCORD_BOT_TOKEN"         # Resolved from environment

voice:
  enabled: true
  stt:
    provider: deepgram
    api_key: "op://MyVault/deepgram/api-key"
  tts:
    provider: deepgram
    api_key: "op://MyVault/deepgram/api-key"

Credentials are resolved once at startup. Plain string values still work for development.

OAuth Token Management

For services requiring OAuth token refresh (Google, Zoom, RingCentral), use the tokens configuration:

# omniagent.yaml
tokens:
  vault_uri: "op://MyVault"
  services:
    google:
      credentials_name: "google-service-account"
      scopes:
        - "https://www.googleapis.com/auth/calendar"
    zoom:
      credentials_name: "zoom-oauth"
    ringcentral:
      credentials_name: "ringcentral-oauth"

The token manager handles:

1. In-memory token caching
2. Automatic refresh when tokens expire
3. Vault coordination for multi-process deployments
4. Refresh token persistence

Vault Environment Variables

Variable	Provider	Description
OP_SERVICE_ACCOUNT_TOKEN	1Password	Service account token (starts with ops_)
BW_ACCESS_TOKEN	Bitwarden	Access token
BW_ORGANIZATION_ID	Bitwarden	Organization ID
BW_API_URL	Bitwarden	Custom API URL (self-hosted)
BW_IDENTITY_URL	Bitwarden	Custom Identity URL (self-hosted)

CLI Commands

# Gateway
omniagent gateway run      # Start the gateway server

# Setup & Diagnostics
omniagent setup            # Interactive setup wizard
omniagent doctor           # Diagnose configuration and connectivity

# Voice (Full-Duplex Phone Calls)
omniagent voice serve      # Start the voice gateway server
omniagent voice status     # Show voice configuration status
omniagent voice call NUM   # Make an outbound call to NUM

# Skills
omniagent skills list      # List all discovered skills
omniagent skills info NAME # Show skill details
omniagent skills check     # Validate skill requirements

# Sessions
omniagent sessions list    # List conversation sessions
omniagent sessions show ID # Show session details
omniagent sessions delete ID # Delete a session

# Channels
omniagent channels list    # List registered channels
omniagent channels status  # Show channel connection status

# OpenAI API
omniagent openai spec      # Generate OpenAPI specification

# Config
omniagent config show      # Display current configuration

# Version
omniagent version          # Show version information

Voice Gateway

Start a full-duplex voice gateway for phone calls:

# Native voice-to-voice (lowest latency, ~100ms)
omniagent voice serve \
  --listen :8081 \
  --public-url https://your-server.com \
  --realtime openai \
  --realtime-voice alloy

# Traditional pipeline (custom STT/TTS)
omniagent voice serve \
  --listen :8081 \
  --public-url https://your-server.com \
  --stt deepgram \
  --tts elevenlabs \
  --llm anthropic \
  --model claude-sonnet-5

Configure Twilio webhooks:

• Voice URL: https://your-server.com/voice/inbound
• Status Callback: https://your-server.com/voice/status

Local Development with ngrok

For local development, use ngrok to expose your local server to Twilio:

# Set ngrok auth token
export NGROK_AUTHTOKEN=your-ngrok-token

# Start voice server with ngrok tunnel (auto-generates public URL)
omniagent voice serve \
  --listen :8081 \
  --ngrok \
  --stt deepgram \
  --tts elevenlabs \
  --llm anthropic

The ngrok public URL will be displayed on startup. Configure this URL in your Twilio webhook settings.

With a custom ngrok domain (requires paid plan):

omniagent voice serve \
  --listen :8081 \
  --ngrok \
  --ngrok-domain myapp.ngrok.io

LiveKit Voice Agents

Run OmniAgent as a voice participant in LiveKit meetings:

# Set credentials
export LIVEKIT_URL="wss://your-project.livekit.cloud"
export LIVEKIT_API_KEY="your-api-key"
export LIVEKIT_API_SECRET="your-api-secret"

# Option 1: Realtime mode (lowest latency, ~100-300ms)
export REALTIME_PROVIDER="deepgram"  # or: openai, gemini
export DEEPGRAM_API_KEY="your-deepgram-key"

# Option 2: Traditional pipeline (STT→LLM→TTS)
# export ANTHROPIC_API_KEY="your-anthropic-key"
# export STT_PROVIDER="deepgram"
# export STT_API_KEY="your-deepgram-key"
# export TTS_PROVIDER="openai"
# export TTS_API_KEY="your-openai-key"

# Run the generic voice agent
go run ./cmd/livekit-agent

# Or run the meeting facilitator (with Meeting PM role)
go run ./cmd/livekit-agent-facilitator

# Or run multi-agent panel discussions
go run ./cmd/livekit-agent-panel

Command	Description
cmd/livekit-agent	Generic voice agent with web search
cmd/livekit-agent-facilitator	Meeting facilitator with Meeting PM role, supports realtime mode
cmd/livekit-agent-panel	Multi-agent panel discussions with HeyGen avatars, JSON scheduling, slides, and recording

Avatar Configuration

Agents can display visual avatars in video tiles. Multiple providers are supported:

Mode	AVATAR_PROVIDER	Description
None	"" (empty)	Audio-only, no video tile
Static	static	Display static image (640x360)
Tavus	tavus	Real-time lip-sync via Tavus
HeyGen	heygen	Real-time lip-sync via HeyGen (panel agents)

# Static image avatar
export AVATAR_PROVIDER="static"
export AVATAR_IMAGE_PATH="./avatar.png"  # Optional, has default

# Tavus live avatar (lip-sync video)
export AVATAR_PROVIDER="tavus"
export TAVUS_API_KEY="your-tavus-key"
export TAVUS_PAL_ID="your-pal-id"  # Optional, uses default

# HeyGen avatars (panel agents)
export HEYGEN_API_KEY="your-api-key"
export HEYGEN_SANDBOX=true  # Use sandbox for testing
export MODERATOR_AVATAR_ID="avatar-id-1"
export PANELIST_1_AVATAR_ID="avatar-id-2"

# Legacy: AGENT_AVATAR=true maps to AVATAR_PROVIDER=static

See omni-livekit avatar documentation and Panel Discussions Guide for detailed setup instructions.

These agents use omni-livekit for LiveKit transport.

Deployment

OmniAgent ships as a single container, built on a digest-pinned Chainguard static runtime image (no shell, no package manager) per the PlexusOne container base-image policy. The Docker Build & Publish GitHub Actions workflow publishes a multi-arch (linux/amd64+linux/arm64) image to ghcr.io/plexusone/omniagent (canonical) and mirrors it to docker.io/grokify/omniagent for discoverability — GHCR pulls remain recommended for production since Docker Hub throttles anonymous pulls — with SBOM and SLSA provenance attestations and a keyless cosign signature on every build (:latest and semver tags on v* releases; :latest + :smoke on a manual run). And OmniDeploy deploys it declaratively to AWS Lightsail via Pulumi:

export AWS_PROFILE="omniagent-omnideploy"   # least-privilege IAM profile
omnideploy up \
    --config deploy/lightsail/deploy.yaml \
    --target lightsail \
    --backend pulumi

• deploy/lightsail/deploy.yaml — tested single-operator (personal-mode) Lightsail config, annotated with the sharp edges found deploying for real. See the Deployment guide for the full walkthrough (IAM policy, GHCR visibility, Pulumi state, secrets).
• deploy/team/prod/ — self-hosted team-mode stack (Caddy + PostgreSQL via Docker Compose on one VM). See the Team Deployment guide.

Architecture

+-------------------------------------------------------------+
|                     Messaging Channels                      |
|     Telegram  |  Discord  |  Slack  |  WhatsApp  |  ...     |
+---------------------------+---------------------------------+
                            |
+---------------------------v---------------------------------+
|              Gateway (WebSocket Control Plane)              |
|              ws://127.0.0.1:18789                           |
+---------------------------+---------------------------------+
                            |
+---------------------------v---------------------------------+
|                      Agent Runtime                          |
|  +------------------+  +------------------+                 |
|  |    Skills        |  |    Sandbox       |                 |
|  |  (SKILL.md)      |  |  (WASM/Docker)   |                 |
|  +------------------+  +------------------+                 |
|  - omnillm (LLM providers)                                  |
|  - omnivoice (STT/TTS)                                      |
|  - omniobserve (tracing)                                    |
|  - Tools (browser, shell, http)                             |
+-------------------------------------------------------------+

Configuration Reference

Gateway

Field	Type	Default	Description
gateway.address	string	127.0.0.1:18789	WebSocket server address
gateway.read_timeout	duration	30s	Read timeout
gateway.write_timeout	duration	30s	Write timeout
gateway.ping_interval	duration	30s	WebSocket ping interval

Agent

Field	Type	Default	Description
agent.provider	string	anthropic	LLM provider
agent.model	string	claude-sonnet-5	Model name
agent.api_key	string	-	API key (or use env var)
agent.temperature	float	unset	Sampling temperature; unset by default so the provider's own default applies — newer Claude models (Sonnet 4.6+/5, Opus 4.6+) reject requests that set temperature at all
agent.max_tokens	int	4096	Max response tokens
agent.system_prompt	string	-	Custom system prompt

Skills

Field	Type	Default	Description
skills.enabled	bool	true	Enable skill loading
skills.paths	[]string	[]	Additional skill directories
skills.disabled	[]string	[]	Skills to skip
skills.max_injected	int	20	Max skills in prompt

Voice

Field	Type	Default	Description
voice.enabled	bool	false	Enable voice processing
voice.response_mode	string	auto	auto, always, never
voice.realtime.provider	string	-	Native voice-to-voice: openai, gemini, deepgram
voice.realtime.voice	string	-	Voice for realtime API
voice.stt.provider	string	-	STT provider (traditional): deepgram, whisper
voice.tts.provider	string	-	TTS provider (traditional): elevenlabs, deepgram

Omni* Library Ecosystem

OmniAgent is built on a modular ecosystem of omni* libraries:

                              OmniAgent
                          (Agent Runtime)
    ┌────────┬────────┬────────┬────────┬────────┬────────┬────────┐
    ▼        ▼        ▼        ▼        ▼        ▼        ▼        ▼
omnichat omnillm omnivoice omniimage omniobserve omniserp omnistorage ...
    │        │        │         │                          │
    │   ┌────┴────┐ ┌─┴──┐      │             ┌────────────┴────────┐
    │   │         │ │    │      │             │                     │
    ▼   ▼         ▼ ▼    ▼      ▼             ▼                     ▼
     omnillm-core  omnivoice-core          omnistorage-core
                                           ├── /object (files)
                                           └── /kvs (sessions)
    │        │        │         │                          │
    └────────┴────────┴─────────┴──────────────────────────┘
                        │
              Provider Modules
    ┌───────────────────┼───────────────────┐
    ▼                   ▼                   ▼
omni-aws           omni-google         omni-github
├── /omnillm       ├── /omnillm        └── /omnistorage
├── /omnistorage   └── /omnistorage
└── /omnivoice

See Architecture Overview for detailed documentation.

Dependencies

Omni* Libraries

Package	Purpose
omnichat	Unified messaging (WhatsApp, Telegram, Discord)
omnillm	Multi-provider LLM abstraction
omnivoice	Voice STT/TTS interfaces
omni-livekit	LiveKit WebRTC voice transport
omni-twilio	Full-duplex voice gateway via Twilio
omni-deepgram	Deepgram STT/TTS and realtime voice
omnimemory	Semantic memory with vector retrieval
omniobserve	LLM observability
omniserp	Web search via Serper/SerpAPI
omniimage	Image generation (OpenAI, Fal AI)
omnistorage-core	Object and key-value storage
omnivault	Secure credential storage
omnitoken

Infrastructure

Package	Purpose
wazero	WASM runtime for sandboxing
moby	Docker SDK for container isolation
Rod	Browser automation
gorilla/websocket	WebSocket server

Related Projects

• omnichat - Unified messaging provider interface
• omnillm - Multi-provider LLM abstraction
• omnivoice - Voice interactions (STT/TTS)
• omni-livekit - LiveKit WebRTC voice transport
• omnimemory - Semantic memory with vector retrieval
• omniimage - Image generation (OpenAI, Fal AI)
• omnirole-facilitator - Meeting PM role
• OpenClaw - Compatible skill format

License

MIT License - see LICENSE for details.