Penny

Your private, personalized internet companion.

Author: Jared Lockhart

Overview

Ask Penny anything and she'll search the web and text you back, always with sources.

But she's not just a question-answering bot. She pays attention. She remembers your conversations, learns what you're into, and starts sharing things she thinks you'd like on her own. She follows up on old topics when she finds something new. She gets to know you over time and her responses get more personal because of it.

Penny communicates via Signal, Discord, iOS, or a Firefox browser extension — all channels share the same conversation history. The browser extension gives her direct access to the web: she can browse pages with the full rendering engine and your session, see what you're currently looking at, and present her discoveries as a browsable feed of thought cards.

Penny is a feed only for you. Private, personal, and local.

How It Works

Conversations

When you send Penny a message, she always searches the web before responding — she never makes things up from model knowledge. A local LLM reads the search results and writes a response in her own voice: casual, calm, with sources. Penny uses the OpenAI Python SDK against any OpenAI-compatible endpoint, so you can run her against Ollama, omlx, the OpenAI cloud, vLLM, or anything else that speaks the protocol.

Penny talks to you over Signal, Discord, an iOS client, or a Firefox sidebar extension — the same apps you already use. All channels share conversation history: ask on Signal, follow up in the browser. Quote-reply to continue a thread; she'll walk the conversation history for context.

Preferences

Penny builds a picture of what you care about. As you chat, she notices the things you bring up with interest or frustration — coffee preferences, weather complaints, hobbies you keep returning to — and files them away as likes and dislikes. A topic has to come up more than once before she takes it seriously, so an offhand comment doesn't send her down a rabbit hole.

You can also just tell her in plain language — "I'm really into dark roast coffee", "I can't stand cold weather", "forget about that one", or "what do you think I'm into?". She files these onto her likes and dislikes memory the same way, no commands to remember. These shape what she thinks about and what she shares with you.

Thinking

When Penny isn't talking to you, she thinks. She picks something you're into, searches the web, and reasons through what she finds — keeping the result as a thought she can come back to. Thoughts feed into future conversations, so when you ask her about something she's been chewing on, the context is already there.

When she finds something genuinely interesting — not a repeat of what she's shared before, and aligned with what you actually like — she sends it to you on her own. She paces herself so you don't get overwhelmed.

Memory

Penny remembers things in a few different shapes — your conversations, the pages she's read, what she's learned about you, the thoughts she's had on her own. When you ask her something, she pulls the most relevant pieces of all of that into context: a page she summarized last week, a topic you mentioned a few days ago, a thought she had this morning. Old material doesn't crowd out the new — only what's actually relevant comes along.

She can also remember things you ask her to. Tell her to keep track of restaurants you've been to, or articles to read later, and she'll start a new memory for it on the spot — no configuration, no code change.

Commands

Beyond regular conversation, Penny supports slash commands:

• /commands — list every command available in this deployment
• /profile — set your name, location, and timezone (required before chat)
• /config — view or tune runtime parameters (30+ values: scheduling intervals, notification backoff, dedup thresholds, email pagination limits, etc.)
• /mute, /unmute — silence or resume autonomous notifications Email is no longer a slash command — ask about your inbox in plain language ("did I get an email from …?", "check my email for …") and Penny drives the email tools (search_emails, read_emails, plus list_emails/list_folders/draft_email on Zoho). The tools are present only when a mailbox is configured — Fastmail via FASTMAIL_API_TOKEN, or Zoho via ZOHO_API_ID/ZOHO_API_SECRET/ZOHO_REFRESH_TOKEN.

Penny's Mind

How information flows through Penny's cognitive systems — from perception to memory to thought to action.

flowchart TB
    User((User)) -->|message| Chat
    User -->|reaction| MemoryFW

    subgraph Conversation["🗣 Conversation"]
        Chat[Chat<br>remembers · browses · responds]
    end

    MemoryFW -.->|"relevant memories"| Chat
    Chat -->|reply| User
    Chat -->|"records the conversation"| MemoryFW

    subgraph MemoryFW["🧠 Memory"]
        Collections["What Penny knows<br>likes · dislikes · pages she's read<br>thoughts · custom memories you've added"]
        Logs["What Penny has heard or done<br>messages · pages browsed"]
    end

    subgraph Background["💭 Background — when idle"]
        Coll["Background work<br>picks an overdue task,<br>updates one memory per cycle"]
    end

    MemoryFW -.->|"cadence + instructions"| Coll
    Coll -->|"writes new entries"| MemoryFW
    Coll -->|"shares thoughts when ready"| User

    style Conversation fill:#e8f5e9,stroke:#2e7d32
    style MemoryFW fill:#e3f2fd,stroke:#1565c0
    style Background fill:#fff3e0,stroke:#e65100

The Cognitive Cycle

1. Conversation — you send a message; Penny pulls the relevant pieces of memory into context (a related page, a recent thought, your preferences), browses the web if she needs to, and responds with sources
2. Background work — when she's idle, Penny works through her to-do list: picking up new preferences from your messages, summarizing pages she's read, generating thoughts on topics you're into, choosing which thoughts are worth sharing. Each task has its own pace
3. Repeat — outgoing messages, browses, and reactions all become memories that future background work can build on

Models

Penny uses up to three LLM model roles, all running locally by default:

Role	Env	Purpose	Required?
Text	LLM_MODEL	Single model for all of Penny's reasoning — chat and background work, including reading photo attachments sent in chat	Yes
Embedding	LLM_EMBEDDING_MODEL	Embeddings for knowledge retrieval, message similarity, and preference dedup	Yes
Image	LLM_IMAGE_MODEL	Image generation (ask Penny to draw — the generate_image tool)	Optional

Text and embedding go through the OpenAI SDK and can each point at a different OpenAI-compatible endpoint via the corresponding LLM_*_API_URL / LLM_*_API_KEY overrides — useful when running text on one machine and embeddings on another. Image generation is the one exception: it talks to Ollama's /api/generate endpoint directly (set LLM_IMAGE_API_URL), because there's no OpenAI-compatible image generation protocol that works with local models. A photo sent in chat rides the chat turn's own user message to the text model, so LLM_MODEL must understand images.

Scheduling

When Penny is idle (default: 60s after the last message) she runs her own background work — extracting preferences, summarizing pages, thinking, choosing what to share. Each collection has its own cadence; Penny picks the most-overdue ready one per tick, and skips the tick entirely if nothing is due. A foreground message cancels any in-progress background task immediately so the LLM is free to respond.

Recurring "watch this and keep me posted" work lives as a collection with a background collector, all managed through the same registry as everything else — there is no separate schedule mechanism.

Runtime Configuration

30+ parameters are tunable at runtime via /config — scheduling intervals, notification backoff, preference dedup thresholds, inner monologue settings, email pagination limits, and more. Values follow a three-tier lookup: database override → environment variable → default. Changes take effect immediately without restart.

Browser Extension

The Firefox extension adds a visual, interactive layer on top of Penny's existing architecture:

• Sidebar chat — same conversation as Signal/Discord, with HTML-formatted responses, images, clickable links, and live in-flight tool status (e.g., "Searching…", "Reading example.com…")
• Active tab context — Penny can see the page you're currently viewing (via Defuddle article extraction, or — on a homepage or section front, which has no article to extract — the page's own list of headlines and where they point). Toggle "Include page content" to ask questions about any page
• Browser tools — browse_url opens pages in hidden tabs with the full web engine and your session. Per-addon "tool use" toggle controls whether each browser participates in tool dispatch
• Domain permissions — first-time access to a new domain triggers an approve/deny prompt. Approvals persist server-side and sync across all connected addons; prompts can also be answered from Signal so you don't need a browser open. /config DOMAIN_PERMISSION_MODE allow_all skips prompting entirely
• Memory Explorer — every memory in one place: a list view with entry counts and last-updated timestamps, plus a drill-in view where you can edit how each memory works (description, what it should remember, how often) and add, edit, or delete individual entries. Each memory's detail page also shows a recent history of background work so you can see exactly what Penny has been doing
• Settings panel — domain allowlist, runtime config params (the same 30+ values /config exposes), and addon-level toggles
• Prompt log viewer — every LLM call Penny makes, browseable from the extension: input prompt, response, internal reasoning, and (for background work) which memory the result was written to and why. Useful for understanding "why did Penny say that"
• Multi-device — each browser registers as a device (e.g., "firefox macbook 16"). All devices share the same user identity and conversation history. In-flight progress reactions on Signal also surface on the user's message via emoji morphing (💭 → 🔍 → 📖 → cleared on completion)

cd browser
npm install
npm run dev    # Build, watch, and launch Firefox with auto-reload

See docs/browser-extension-architecture.md for the full architecture and security model.

iOS Channel

The iOS channel is a foreground WebSocket plus APNs preview-notification path:

• Foreground app — the iOS app connects to Penny's WebSocket, registers its device/APNs token, sends chat messages, and pulls queued outbox rows. When Penny has a new message while the socket is connected, she sends an outbox_changed hint and the app pulls messages.
• Background/offline app — Penny still writes every outgoing message to the durable iOS outbox. If the device is not connected, Penny sends an APNs alert titled Hi from Penny. Its body includes as many consecutive opening sentences as fit within PUSH_PREVIEW_CHARS (450 characters by default), after markup is removed. If the first sentence alone is too long, the body is truncated with an ellipsis inside the limit. Source metadata, such as Notifier, Chat, or Collector: flight-deals, accompanies the queued message.
• Live updates — Prompt logs, run outcomes, and memory changes are broadcast independently to connected devices. A device disconnecting during an update is handled without an unobserved background-task error.
• Reconnect — the app sends pull_messages, displays/persists the returned rows, then sends ack_messages so Penny can mark them delivered.

Device registration is gated by IOS_PAIRING_TOKEN when configured. The client sends a register WebSocket message with device_id, label, pairing_token, optional device_secret, apns_token, apns_environment, and app_version. When Penny is running with CHANNEL_TYPE=ios, the registered iOS device becomes the default channel target. When IOS_ENABLED=true is used alongside Signal, iOS runs in parallel while Signal remains the default target until the default device is changed.

For APNs setup, create an Apple Push Notifications authentication key in the Apple Developer portal. IOS_APNS_KEY_ID is the Key ID shown for that key, and IOS_APNS_KEY_PATH should point at the downloaded .p8 file inside the container. The repo mounts ./data at /penny/data, so a common setup is to put the key under data/private/AuthKey_XXXX.p8 and set IOS_APNS_KEY_PATH="/penny/data/private/AuthKey_XXXX.p8". .p8 files are ignored by git.

If production APNs tokens should use a different auth key, also set IOS_APNS_PRODUCTION_TEAM_ID, IOS_APNS_PRODUCTION_KEY_ID, and IOS_APNS_PRODUCTION_KEY_PATH. Set IOS_APNS_PRODUCTION_BUNDLE_ID when the production APNs topic differs from IOS_BUNDLE_ID, such as a TestFlight bundle ID. When an iOS client registers with apns_environment=production, Penny uses those production credentials; otherwise it falls back to the default APNs credentials above.

In the iOS app's Connection settings, tap Send beside Test Notification to force a test APNs notification to that registered device, even while the WebSocket is connected. The app sends a test_notification WebSocket message for this diagnostic.

iOS Service Diagram

Example service diagram for an iOS setup, using Openrouter as a model host and a Digital Ocean droplet running Nginx as a proxy.

Setup & Running

Prerequisites

1. For Signal: signal-cli-rest-api running on host (port 8080)
2. For Discord: Discord bot token and channel ID
3. For iOS: an iOS client that speaks the Penny WebSocket protocol; APNs credentials if you want background notifications
4. An OpenAI-compatible LLM endpoint running on host or reachable from the container. Set LLM_API_URL to point at it. Common choices: Ollama, omlx, vLLM, the OpenAI cloud
5. Browser extension loaded in Firefox (for web search, page reading, and the visual UI)
6. Docker & Docker Compose installed

Quick Start

# 1. Create .env file with your configuration
cp .env.example .env
# Edit .env with your settings (Signal or Discord credentials)

# 2. Start the agent
make up

Make Commands

make up               # Build and start Penny (foreground, with dev source mounts)
make prod             # Deploy penny only (no dev override)
make prod-ios         # Run Penny as the iOS channel without starting signal-api
make kill             # Tear down containers and remove local images
make docker-prune     # Global best-effort reclaim: prune stopped containers, dangling images, build cache, unused volumes
make build            # Build the penny Docker image
make browser-build    # Bundle the browser extension content script
make check            # Format check, lint, typecheck, migrate-validate, eval cold imports, eval-catalogue declarations, pytest, tsc (browser)
make pytest           # Run integration tests
make eval-remote      # Live-model eval suite on a remote provider, pinned to one, 8 cases x 5 samples at once
make eval-report      # Post a completed eval run's assembled report to its PR
make fix              # Format + autofix lint issues
make catalogue        # Generate docs/eval-catalogue.md (gitignored, never committed) from the eval cases
make typecheck        # Type check with ty
make token            # Generate GitHub App installation token for gh CLI
make signal-avatar    # Set Penny's Signal profile picture
make migrate-test     # Test database migrations against a copy of prod DB
make migrate-validate # Check for duplicate migration number prefixes
make client-check     # Build the iOS client + run its tests on a simulator (requires Xcode)

All dev tool commands run in temporary Docker containers via docker compose run --rm, with source volume-mounted so changes write back to the host filesystem.

Configuration is managed via a .env file in the project root:

# .env

# Channel type (optional — auto-detected from credentials)
# CHANNEL_TYPE="signal"  # "signal", "discord", or "ios"

# Signal Configuration (required for Signal)
SIGNAL_NUMBER="+1234567890"
SIGNAL_API_URL="http://localhost:8080"

# Discord Configuration (required for Discord)
DISCORD_BOT_TOKEN="your-bot-token"
DISCORD_CHANNEL_ID="your-channel-id"

# Browser Extension (optional)
BROWSER_ENABLED=true
BROWSER_HOST="0.0.0.0"                    # Use 0.0.0.0 in Docker
BROWSER_PORT=9090

# iOS Channel (optional)
IOS_ENABLED=false
IOS_HOST="0.0.0.0"                        # Use 0.0.0.0 in Docker
IOS_PORT=9091
IOS_PAIRING_TOKEN=""                      # Required by server if set; client sends it in register
IOS_APNS_TEAM_ID=""                       # Apple Developer Team ID
IOS_APNS_KEY_ID=""                        # APNs auth key ID
IOS_APNS_KEY_PATH=""                      # Container path to .p8, e.g. /penny/data/private/AuthKey_XXXX.p8
IOS_APNS_PRODUCTION_TEAM_ID=""            # Optional production APNs Team ID override
IOS_APNS_PRODUCTION_KEY_ID=""             # Optional production APNs auth key ID override
IOS_APNS_PRODUCTION_KEY_PATH=""           # Optional production APNs .p8 path override
IOS_BUNDLE_ID=""                          # iOS app bundle id, e.g. com.example.Penny
IOS_APNS_PRODUCTION_BUNDLE_ID=""          # Optional production APNs topic, e.g. TestFlight bundle id
IOS_APNS_SANDBOX=true                     # Fallback only: devices report sandbox/production at registration

# LLM Configuration — any OpenAI-compatible endpoint (Ollama, omlx, vLLM,
# OpenAI cloud, ...). The example URL points at a local Ollama instance.
LLM_API_URL="http://host.docker.internal:11434/v1"
LLM_MODEL="gpt-oss:20b"                   # Single model for all agents — reads photo attachments in chat
# LLM_API_KEY="not-needed"                # Default fine for unauthenticated local backends
LLM_EMBEDDING_MODEL="embeddinggemma"      # Required — backs memory (preference dedup + similarity recall)
# LLM_IMAGE_MODEL="x/z-image-turbo"       # Optional, enables the generate_image tool (uses LLM_IMAGE_API_URL)
# LLM_IMAGE_API_URL="http://host.docker.internal:11434"  # Ollama REST for image generation

# Database & Logging
DB_PATH="/penny/data/penny/penny.db"
LOG_LEVEL="INFO"
# LOG_FILE="/penny/data/penny/logs/penny.log"

# Fastmail JMAP (optional, enables the email tools on the chat surface)
# FASTMAIL_API_TOKEN="your-api-token"

# Zoho Mail (optional, enables the email tools on the chat surface, Zoho backend)
# ZOHO_API_ID="..."
# ZOHO_API_SECRET="..."
# ZOHO_REFRESH_TOKEN="..."

# GitHub App (optional, used by `make token` for authenticated gh CLI access)
# GITHUB_APP_ID="12345"
# GITHUB_APP_PRIVATE_KEY_PATH="data/private/github-app.pem"
# GITHUB_APP_INSTALLATION_ID="67890"

Channel Selection

Penny auto-detects which channel to use based on configured credentials:

• If DISCORD_BOT_TOKEN and DISCORD_CHANNEL_ID are set (and Signal is not), uses Discord
• If SIGNAL_NUMBER is set, uses Signal
• If IOS_ENABLED=true is set with Signal, the iOS WebSocket/APNs channel starts in parallel and Signal remains the primary/default channel
• If only IOS_ENABLED=true is set, uses iOS
• Set CHANNEL_TYPE explicitly to override auto-detection. Use CHANNEL_TYPE=ios to make iOS the primary/default channel without starting Signal.
• Use make prod-ios to run Penny with CHANNEL_TYPE=ios without starting the Signal container.
• The signal-api container only starts when SIGNAL_NUMBER is set. make up/make prod enable the signal compose profile from that same value, so a Discord (or iOS) deployment runs Penny alone — it never launches signal-api and never blocks on it at startup. Signal deployments still wait for signal-api's healthcheck (the cold-boot race is unchanged).

Configuration Reference

LLM — Penny talks to any OpenAI-compatible endpoint via the OpenAI Python SDK. There are no Ollama-specific dependencies in the runtime.

• LLM_API_URL: API endpoint (default: http://host.docker.internal:11434)
• LLM_MODEL: Single text model for all agents (default: gpt-oss:20b). Photo attachments ride the chat turn's own user message to this model, so it must understand images
• LLM_API_KEY: API key (default: "not-needed", fine for unauthenticated local backends)
• LLM_EMBEDDING_MODEL: Required. Dedicated embedding model (e.g., embeddinggemma) that backs preference/knowledge/message embeddings — Penny fails fast at startup if it is unset
• LLM_EMBEDDING_API_URL / LLM_EMBEDDING_API_KEY: Override API URL/key for the embedding model
• LLM_IMAGE_MODEL: Image generation model (e.g., x/z-image-turbo). Optional; enables the generate_image chat tool (ask Penny to draw). Image generation is the one non-OpenAI endpoint — it talks to Ollama's /api/generate directly
• LLM_IMAGE_API_URL: Ollama REST endpoint for image generation (default: http://host.docker.internal:11434)

API Keys:

• FASTMAIL_API_TOKEN: enables the email tools on the chat surface (Fastmail)
• ZOHO_API_ID, ZOHO_API_SECRET, ZOHO_REFRESH_TOKEN: enables the email tools on the chat surface (Zoho backend; obtain via Zoho's OAuth flow)

GitHub App (required for agent containers):

• GITHUB_APP_ID, GITHUB_APP_PRIVATE_KEY_PATH, GITHUB_APP_INSTALLATION_ID

Browser Extension (optional):

• BROWSER_ENABLED: true to start the WebSocket server (default: false)
• BROWSER_HOST: bind address (default: localhost; use 0.0.0.0 in Docker)
• BROWSER_PORT: WebSocket port (default: 9090)

iOS Channel (optional):

• IOS_ENABLED: true to enable the iOS channel alongside the auto-detected primary channel; with Signal configured, both Signal and iOS listen in parallel
• IOS_HOST: WebSocket bind address (default: 0.0.0.0, the Docker default; use localhost for host-only binding)
• IOS_PORT: WebSocket port (default: 9091)
• IOS_PAIRING_TOKEN: optional shared token required during device registration when set
• IOS_APNS_TEAM_ID: Apple Developer Team ID
• IOS_APNS_KEY_ID: APNs auth key ID from the Apple Developer portal
• IOS_APNS_KEY_PATH: container path to the APNs .p8 auth key
• IOS_APNS_PRODUCTION_TEAM_ID: optional production APNs Team ID override
• IOS_APNS_PRODUCTION_KEY_ID: optional production APNs auth key ID override
• IOS_APNS_PRODUCTION_KEY_PATH: optional container path to the production APNs .p8 auth key
• IOS_BUNDLE_ID: iOS app bundle identifier used as the APNs topic
• IOS_APNS_PRODUCTION_BUNDLE_ID: optional production APNs topic override, for example the TestFlight bundle identifier
• IOS_APNS_SANDBOX: fallback APNs environment (true = sandbox) for devices that did not report a recognized apns_environment at registration; the per-device value wins otherwise

Behavior:

• TOOL_TIMEOUT: Tool execution timeout in seconds (default: 120)
• MAX_STEPS / IDLE_SECONDS: also accepted as env vars, but these are runtime-configurable via /config so DB overrides win
• 30+ parameters are runtime-configurable via /config — scheduling intervals, notification cooldowns/candidates, preference dedup thresholds, history context limits, email body/search/list pagination limits, related-message retrieval thresholds, and more

Logging:

• LOG_LEVEL: DEBUG, INFO, WARNING, ERROR (default: INFO)
• LOG_FILE: Optional path to log file
• LOG_MAX_BYTES: Maximum log file size before rotation (default: 10 MB)
• LOG_BACKUP_COUNT: Number of rotated backup files to keep (default: 5)
• DB_PATH: SQLite database location (default: /penny/data/penny/penny.db)

1. Create a Discord application at https://discord.com/developers/applications
2. Create a bot for the application and copy the token
3. Enable these intents in the Bot settings under Privileged Gateway Intents:
   • Message Content Intent — required. Without it Discord refuses the gateway connection and Penny exits at startup with an actionable log line pointing you here.
   • Server Members Intent (optional)
4. Invite the bot to your server with the OAuth2 URL Generator:
   • Scopes: bot
   • Permissions: Send Messages, Read Message History
5. Get the channel ID (enable Developer Mode in Discord settings, right-click channel → Copy ID)
6. Add to your .env:

   DISCORD_BOT_TOKEN="your-token"
   DISCORD_CHANNEL_ID="your-channel-id"
   

Penny includes end-to-end integration tests that mock all external services:

make pytest      # Run all tests
make check       # Run format, lint, typecheck, and tests

CI runs make check in Docker on every push to main and on pull requests via GitHub Actions. Pull requests touching penny-client/ also run make client-check on a macOS runner, building the iOS app and running its test suite on a simulator.

Tests cover the full message flow (search, response, threading, typing indicators), all background agents (history, thinking, notify, scheduler coordination), every slash command, image handling, and tool edge cases. External services are replaced with mock servers and SDK patches — a mock Signal WebSocket server and a mock LLM client (MockLlmClient, patches openai.AsyncOpenAI) with configurable responses.

License

MIT