vimmary

Summary service for YouTube videos and podcast episodes. Fetches video transcripts via YouTube's InnerTube API and podcast transcripts from cast2md, generates LLM summaries, and stores everything in Postgres + pgvector for semantic search. Videos can be added manually via the web UI or automatically through Karakeep webhooks; podcast episodes arrive through per-feed subscriptions or a deep link from cast2md.

Videos and podcasts stay separated in the UI and in RSS. GET /api/v1/videos defaults to videos only, and the original feed URL keeps serving videos alone — the combined view is an explicit third option, never the default.

How it works

Karakeep ──webhook──▶ vimmary ──▶ fetch transcript ──▶ generate summary
Web UI ──manual URL──▶    │       (InnerTube / cast2md)        │
cast2md ──poll/deep-link─▶│                     ┌─────────────┼──────────────┐
                          │                     ▼             ▼              ▼
                          │                 pgvector      Karakeep        Web UI
                          │                 + search      writeback       display
                          │                 (videos only)                    │
                          ◀──── MCP tools ──────────────────────────────────┘

1. A YouTube video is bookmarked in Karakeep (webhook) or submitted manually via the web UI; a podcast episode is picked up by the cast2md poller or sent over from cast2md's episode page
2. vimmary fetches the transcript — via YouTube's InnerTube API for videos, from cast2md for episodes
3. An LLM (Claude or Mistral) generates a structured summary, using a prompt written for the source
4. The summary is stored with embeddings for semantic search
5. Results are written back to Karakeep (videos only) and displayed in the web UI

Features

• Manual URL submission — paste any YouTube URL in the web UI to process it immediately
• Automatic summaries — triggered by Karakeep webhooks, no manual action needed
• Bulk import — import all existing YouTube bookmarks from Karakeep via Settings page
• Two detail levels — medium (automatic) and deep (on-demand via MCP or web UI)
• Hybrid search — keyword + semantic search with Reciprocal Rank Fusion
• Adaptive rate limiting — YouTube API delays scale with queue depth (10s–45s) to avoid 429s during bulk operations
• Auto-retry — transcript fetch failures are automatically retried with exponential backoff (2m/5m/10m, max 3 retries)
• Retry all failed — batch-retry all failed videos from the web UI
• Podcast summaries — subscribe to cast2md feeds per user; every episode transcribed from then on is summarized automatically, with per-source prompts and a separate worker queue
• Podcast backfill — subscribing summarizes the 3 newest episodes by default; per feed you can pull the last N, summarize the whole back catalogue, or ask cast2md to transcribe the rest of the feed. None of it disturbs the running poll.
• MCP server — 5 tools for searching and browsing summaries of both kinds
• RSS feeds — three Atom feeds per user (videos, podcasts, both), with per-user feed tokens for authentication
• Web UI — React frontend embedded in the Go binary (Videos, Podcasts, Stats, Settings pages)
• Tailscale auth — zero-config authentication via tsnet
• Multi-user support — per-user video libraries (same YouTube video can be bookmarked by multiple users independently)
• Per-user Karakeep integration — each user configures their own API key and webhook token via the Settings page
• Bidirectional sync — summaries written back to Karakeep notes; bookmark deletions in Karakeep remove videos from vimmary
• Karakeep writeback — plain-text summary with vimmary detail link, video-summarized tag added (preserves existing Karakeep AI tags)

Scope

vimmary targets classic YouTube videos — talks, tutorials, conference recordings — and podcast episodes that cast2md has already transcribed. Livestreams, Shorts and playlists are deliberately not supported: their transcripts are either absent or too unstructured for a useful summary.

vimmary does not download or transcribe audio for podcasts. That is cast2md's job; vimmary reads the finished transcript over the tailnet.

It is sized for personal use, roughly ten videos per day at the upper end. That assumption shows up in a few places on purpose — the feed serves 50 entries without paging, there is no conditional GET, and processing runs through a single worker. Summaries are primarily English with occasional German.

Architecture

Component	Technology
Backend	Go, chi router
Database	PostgreSQL 16 + pgvector
Embeddings	Mistral (mistral-embed, 1024-dim)
Summaries	Claude API or Mistral (configurable)
Auth	Tailscale tsnet
Secrets	API keys in the database, DB password from the environment
Transcripts	YouTube InnerTube API (native Go)
Search	Hybrid: keyword + semantic with RRF
MCP	mcp-go, HTTP + stdio transports
Frontend	React + Vite (embedded in Go binary)

Quick start

Prerequisites

• Docker and Docker Compose
• A Mistral API key for embeddings (console.mistral.ai)
• A Claude API key or Mistral API key for summaries

1. Create a project directory

mkdir vimmary && cd vimmary

2. Fetch the compose file

curl -LO https://raw.githubusercontent.com/meltforce/vimmary/main/compose.example.yml

It runs ghcr.io/meltforce/vimmary:latest alongside pgvector. (docker-compose.yml in the repo is the development stack — it pulls :edge from a tailnet-internal registry and is not usable from outside.)

3. Create config.yaml

external_url: "http://localhost:8080"

server:
  host: "0.0.0.0"
  port: 8080

database:
  host: db
  port: 5432
  name: vimmary
  user: vimmary

summary:
  provider: "claude"          # "claude" or "mistral"

youtube:
  sub_langs: [en]             # preferred transcript languages

secrets:
  postgres_password: "vimmary"
  mistral_api_key: "your-mistral-key"   # required (embeddings)
  claude_api_key: "your-claude-key"     # required if provider is "claude"

Host, port and database settings can be overridden from the environment: VIMMARY_SERVER_HOST, VIMMARY_SERVER_PORT, VIMMARY_DB_HOST, VIMMARY_DB_PORT, VIMMARY_DB_NAME, VIMMARY_DB_USER, VIMMARY_DB_SSLMODE, VIMMARY_TS_ENABLED, VIMMARY_TS_HOSTNAME, VIMMARY_TS_STATE_DIR. Secrets are read from config.yaml or from the configured secret backend, not from the environment.

4. Start

# Postgres reads its password from .env; it has to match secrets.postgres_password
echo "POSTGRES_PASSWORD=vimmary" > .env
docker compose -f compose.example.yml up -d

Open http://localhost:8080 and start adding videos. Migrations run automatically on startup.

Local development

# Start only the database
docker compose up db

# Run the backend (requires Go 1.23+)
go run ./cmd/vimmary --config config.yaml

# Run the frontend with hot-reload (separate terminal)
cd web && npm install && npm run dev

Setup Karakeep integration

1. Open vimmary's Settings page (Tailscale auth required)
2. Enter your Karakeep API key (from Karakeep Settings → API Keys)
3. Copy the generated Webhook URL and Bearer Token
4. In Karakeep Settings → Webhooks, create webhooks for created and deleted events
5. If Karakeep runs in Docker and vimmary is on Tailscale, add CRAWLER_ALLOWED_INTERNAL_HOSTNAMES=.your-tailnet.ts.net to Karakeep's env to allow webhook delivery

Submit a video without Karakeep

GET /submit?url=<video URL> queues a video directly. The route sits behind the Tailscale identity middleware, so the calling device's Tailscale login is the whole authentication — no token, and the summary lands on that user's row. It answers a redirect to the list, which makes it usable as a share-sheet target, an iOS Shortcut, or a bookmarklet:

https://vimmary.<tailnet>.ts.net/submit?url=https://youtu.be/dQw4w9WgXcQ

Nothing is written back to Karakeep for a video submitted this way, because there is no bookmark to write to.

The web UI's own form posts {"url": "..."} to POST /api/v1/videos, which is the same path for a JSON client.

Setup podcast summaries

Podcast transcripts come from cast2md. vimmary polls it; cast2md never calls vimmary. Both services must be on the same tailnet, because cast2md has no authentication and is reachable only there.

1. In vimmary's config, set cast2md.enabled: true and cast2md.base_url to the cast2md host
2. Restart vimmary. Its poller starts 30 seconds later and does nothing until a feed is subscribed
3. Open Settings → Podcasts, tick the feeds you want and pick a detail level per feed

Subscribing summarizes the feed's 3 newest transcribed episodes and follows along from there. That first poll runs when you tick the box, not at the next tick, so the summaries are queued within seconds. The number is per feed — on subscribe: none restores plain "from now on", where the first poll only records a watermark. Either way the watermark ends up at the newest episode of that first batch, so nothing is summarized twice. Switching a feed off keeps its watermark, so switching it back on later fetches the gap through the ordinary poll.

Three ways to reach older episodes, none of which moves the watermark:

Action	Does
Backfill (last 5/10/25/50)	Summarizes that many of the newest transcribed episodes.
Summarize all	Summarizes every episode cast2md has a transcript for. Only model calls.
Transcribe all	Asks cast2md to download and transcribe the rest of the feed. They appear here as they finish, through the ordinary poll — but only if the feed is subscribed.

Summarize all and Transcribe all each state their episode count and ask before running. Transcribe all is the one action that makes cast2md do work rather than just reading from it; on a large feed it can mean hundreds of downloads and Whisper runs.

To summarize one episode without subscribing, set vimmary_url in cast2md's settings. Completed episodes then carry a Summarize in vimmary button that links to vimmary's /podcasts/new?episode=<id> page.

Podcasts have their own prompts, editable per level under Settings → Summaries with the Videos/Podcasts switch. Editing one does not touch the other.

RSS feeds

vimmary provides Atom feeds of your summaries, including full summaries, key points, and action items. Each entry links back to the vimmary summary page and to the source — YouTube for videos, the cast2md episode page for podcasts.

There are three feeds behind one token:

URL	Contents
/feed/atom/<feed-token>	Videos only. This is the original URL, so existing subscriptions are unaffected by podcast summaries appearing.
/feed/atom/<feed-token>/podcasts	Podcast episodes only.
/feed/atom/<feed-token>/all	Both. Every entry carries its type as its first <category>.

An RSS reader cannot filter, which is why the split lives in the URL rather than in a query parameter.

1. Open vimmary's Settings page
2. Copy the URL of the feed you want from the RSS section (the token is generated automatically on first access)
3. Subscribe in your RSS reader
4. Optional: append ?limit=100 to fetch more than the default 50 entries (max 200)

Each user has their own feed token. The token is the only authentication — no Tailscale auth is needed for the feed URL, so it works with any RSS reader.

MCP configuration

The MCP server is always available at /mcp (HTTP + SSE transport) and can also be started in stdio mode via --mcp flag for local use.

HTTP (production): Add vimmary as an MCP server in your client using the SSE endpoint:

{
  "mcpServers": {
    "vimmary": {
      "url": "https://<your-vimmary-host>/mcp"
    }
  }
}

Stdio (local development):

{
  "mcpServers": {
    "vimmary": {
      "command": "go",
      "args": ["run", "./cmd/vimmary", "--mcp", "--config", "config.yaml"]
    }
  }
}

Authentication is handled via Tailscale (HTTP mode) or defaults to user ID 1 (stdio mode).

Build

# Build binary
CGO_ENABLED=0 go build -o vimmary ./cmd/vimmary

# Build Docker image
docker buildx build --platform linux/amd64 -t vimmary:local .

MCP tools

Tool	Description
search_videos	Hybrid search (keyword + semantic, RRF)
get_video	Retrieve full video details by ID
list_recent	Browse recent videos with filters
resummarize	Regenerate summary with different detail level
stats	Aggregate statistics, optionally per source

Repository documents

File	Holds
CLAUDE.md	What the repo is, and the gotchas the file tree does not show.
DECISIONS.md	Decisions taken, with their reasoning and the condition that would re-open them.
ROADMAP.md	Open work only.
INCIDENTS.md	Postmortems.

Related projects

• meltkit — shared Go library (db, config, secrets, middleware, MCP)
• totalrecall — personal knowledge system (architectural blueprint)

---

Source-of-truth development happens on a self-hosted Forgejo (private); github.com/meltforce/vimmary is the public mirror.