VideoLens

Turn a long video into a useful written report with the important ideas, visual context, and timestamped evidence preserved.

VideoLens is a broad video-to-report tool with a simple YouTube-first web experience. Paste a link, choose a report style, and get a detailed artifact you can read, search, save, or question later. Unlike transcript-only summarizers, VideoLens combines speech, sampled frames, and on-screen text before producing a professional standalone HTML report, print-quality PDF, Markdown, and machine-readable JSON. Local files and other supported video URLs work too.

View the no-key sample report · Analyze a YouTube video

videolens analyze "https://www.youtube.com/watch?v=..." \
  --mode general \
  --prompt "Write a detailed report with the main ideas, evidence, and takeaways."

Or launch the web UI:

videolens ui

---

Why VideoLens

Most video summarizers stop at the transcript. That misses slides, diagrams, demonstrations, code, charts, and other information carried visually. VideoLens builds a time-aligned record of what was said and shown, then writes a structured report with citations back to the important moments. A cached intermediate timeline also lets people and agents ask more questions without repeating the expensive extraction.

Status

v0.1 — alpha. End-to-end working: local files + direct URLs + YouTube → timeline → mode-driven analysis → HTML/PDF/Markdown/JSON reports. UI included. Not production-tested at scale.

Features

• Multi-source input — local files (.mp4 / .mov / .webm / .mkv), direct video URLs, YouTube
• Cloud-only pipeline — OpenAI for transcription, frame description (+ OCR), and synthesis
• Timestamped timeline — every finding cites evidence at a specific second of the video
• Four report starting points in the web app: Detailed Report, Key Insights, Tutorial Guide, and Interview / Podcast
• Professional report export — standalone branded HTML and a matching print-quality PDF
• Nine underlying analysis modes for specialized use: general, bug, meeting, ux, tutorial, product_demo, content, privacy, and production_recipe
• Local web UI (Streamlit) — drag-and-drop upload, mode/frame sliders, tabbed results (Report / Timeline / Frames / Transcript / Cache)
• Per-step caching — re-runs are cheap; per-prompt analysis cache means "ask many questions of one extraction" is free after the first
• Cost-aware frame budgeting — --max-frames caps vision-API calls; adaptive interval

Architecture

Source URL or file
   │
   ▼
┌──────────┐   classifies local_file / youtube / direct_url / webpage,
│ resolver │   reports limitations clearly
└──────────┘
   │
   ▼
┌────────────┐   yt-dlp (when remote), ffprobe metadata, ffmpeg audio
│ processors │   extraction + 30 s chunking, ffmpeg frame sampling,
└────────────┘   OpenAI transcription (parallel per chunk), OpenAI vision
   │             frame describe + OCR combined (parallel per frame)
   ▼
┌──────────┐   merges frame summaries + transcript into time-windowed
│ timeline │   segments; per-segment OCR, visual, transcript, confidence
└──────────┘
   │
   ▼
┌──────────┐   one GPT-5.5 call with timeline + user prompt + mode
│ analysis │   guidance → structured Analysis (summary, findings with
└──────────┘   timestamped evidence, recommendations, tasks, limitations)
   │
   ▼
┌──────────┐
│ outputs  │   report.html + report.md + analysis.json + cached artifacts
└──────────┘

Each step is cached at .videolens/cache/<source-hash>/ so re-runs are cheap. Cache key includes source content (hashed) plus extraction settings (frame interval, max frames).

Install

Requires Python 3.12+ and ffmpeg + ffprobe on PATH.

# Install ffmpeg (macOS)
brew install ffmpeg

# Clone and install with uv
git clone https://github.com/shadoprizm/videolens.git
cd videolens
uv sync --extra ui

Set your OpenAI API key:

export OPENAI_API_KEY=sk-...

(The web UI also accepts the key as a password-style input if you'd rather not put it in your environment.)

Usage

Web UI (recommended)

uv run videolens ui

Opens http://localhost:8501. Drop a video, set a prompt and mode, hit Analyze.

Vercel deployment

Vercel runs the root FastAPI entrypoint declared in pyproject.toml:

[tool.vercel]
entrypoint = "app:app"

The Vercel deployment exposes a lightweight HTTP surface at / and /api/health. The full upload/analyze Streamlit UI remains a local app launched with uv run videolens ui. Keeping the UI dependencies in the optional ui extra prevents Vercel's Python function bundle from pulling in Streamlit, pandas, pyarrow, and PDF rendering packages.

Railway deployment

Railway is the recommended hosted path for the full Streamlit UI. The repo includes:

• Dockerfile — installs Python dependencies, ffmpeg, and PDF-rendering system libraries
• railway.json — tells Railway to use the Dockerfile and health-check Streamlit
• .dockerignore — keeps local caches, videos, and virtualenvs out of the image

In Railway:

1. Create a new project from github.com/shadoprizm/videolens.
2. Deploy. Railway will build with the root Dockerfile.
3. In Networking, generate a public domain with target port 8501.

Do not set a shared OPENAI_API_KEY for the hosted app. VideoLens is BYOK: users paste their own OpenAI key into the UI, use the tool, and the key is kept only in that browser session.

Optional: add a Railway volume mounted at /app/.videolens if you want uploads and processed-video cache files to survive redeploys.

CLI

uv run videolens analyze <source> --prompt "<your question>" [--mode MODE]

Useful flags:

Flag	Default	Purpose
--mode, -m	general	Analysis mode
--max-frames	40	Hard cap on frames sent to the vision model (cost dial)
--frame-interval	5.0	Seconds between sampled frames (adaptive — grows if needed to respect max-frames)
--output-dir, -o	./output/videolens	Where report.html + report.md + analysis.json are written
--force	off	Bypass cache and reprocess
--json	off	Skip terminal summary printout
--verbose, -v	off	Verbose logging

Examples

# Turn a YouTube video into a detailed written report
videolens analyze "https://www.youtube.com/watch?v=..." --mode general \
  --prompt "Write a thorough report with the important ideas, examples, caveats, and conclusions."

# Convert a tutorial into a guide
videolens analyze "https://www.youtube.com/watch?v=..." --mode tutorial \
  --prompt "Create an ordered guide with prerequisites, exact steps, warnings, and verification checks."

# Extract only the information worth remembering
videolens analyze "https://www.youtube.com/watch?v=..." \
  --prompt "Extract the key ideas, facts, examples, and conclusions without repetition or filler."

Modes in v0.1

Mode	Use it for
general	Broad review: what's happening, what's notable, what's worth knowing
bug	Bug recordings → repro steps, severity hint, ticket-ready summary
meeting	Calls/standups/briefings → decisions, objections, commitments, follow-ups (uses diarized transcription)
ux	Session replays → user intent, friction, abandoned flows, and product fixes
tutorial	How-to videos → ordered steps, prerequisites, commands, warnings, and a checklist
product_demo	Product demos → feature inventory, positioning, proof, gaps, and opportunities
content	Video content → hook, pacing, clarity, proof, editing opportunities, and CTA
privacy	Share-ready review → possible credentials, personal data, internal URLs, and redaction plan
production_recipe	Reference videos → how the video itself was made: script spine, shot inventory, edit rhythm, likely tools, asset checklist, and recreation recipe

Each mode is a small prompt-fragment file under src/videolens/analysis/modes/. Adding a new mode is ~30 lines.

Models (configurable)

Defaults are in src/videolens/config.py:

Stage	Model
Transcription (general/bug)	gpt-4o-mini-transcribe
Transcription (meeting)	gpt-4o-transcribe-diarize
Prompt enhancement + frame OCR	gpt-5.6-terra (no reasoning)
Final analysis + follow-up Q&A	gpt-5.6-terra (medium reasoning)

Swap the model IDs, reasoning effort, and image detail by editing the Models dataclass. Frames are bounded to 1920x1080 before using original-detail vision input, preserving OCR quality without letting 4K uploads create unbounded image-token costs.

Cost guide

Rough order-of-magnitude:

• A 30-second video with 5 frames → typically under $0.15
• A 5-minute video with 20 frames → typically under $0.50
• A 30-minute meeting with 40 frames → typically $0.75–$2.00

Per-prompt analysis cache means asking a second question of an already-extracted video costs only the synthesis call (cents).

Session replays (PostHog / Hotjar / Clarity / FullStory / LogRocket / OpenReplay)

These services store user-session events, not video — yt-dlp can't extract them. VideoLens ships an optional browser-capture fallback that opens the replay URL in headless Chromium, starts playback, and records the viewport in real time:

uv sync --extra capture
uv run playwright install chromium

After installation, paste any PostHog/Hotjar/etc. share URL and VideoLens routes it through Chromium automatically. Tune the recording window via --capture-duration <seconds> on the CLI (default 60s).

Self-host only for now. Real-time capture takes as long as the replay itself, which conflicts with the request lifecycle of the managed-PaaS deployment at app.videolens.io. The hosted instance falls back to the existing "screen-record manually and upload" workflow for replay services.

MCP server (Claude Code / Cursor / Windsurf)

VideoLens ships a Model Context Protocol server so AI agents can analyse videos as a first-class tool.

uv sync --extra mcp

Available tools:

Tool	What it does
analyze_video	Run the full pipeline against a source + prompt + mode
ask_video	Follow-up question against a previously analysed video
get_timeline	Fetch the cached timeline (segments with visual/OCR/transcript)
get_transcript	Fetch the cached transcript
get_frames	Fetch frame summaries
list_cached	List all videos with cached extraction artifacts

Add it to Claude Code:

claude mcp add videolens -- videolens-mcp

Or wire it into Cursor / Windsurf / any MCP-aware host the same way as any stdio MCP server.

Chrome extension

The VideoLens side-panel extension offers two explicit processing modes:

• Private / BYOK (free forever): the pipeline runs inside the browser and calls OpenAI directly with the user's key. VideoLens receives no analysis content.
• Pro / Managed: the user connects a VideoLens account, no OpenAI key is required, and managed calls run through Vercel AI Gateway to the configured provider (currently OpenAI) while counting against the account allowance. Completed reports enter the cloud library only when the user enables cloud saving.

Both modes analyze the current tab (YouTube captions + canvas frame sampling) or a local file (in-browser audio decode → transcription). The OpenAI key used by Private mode never leaves the device.

• Free and open source Private mode — unlimited analyses and no VideoLens account
• Three managed starter reports for signed-in users; Pro includes 20 managed reports per calendar month
• Four reader-focused report styles plus 6 specialized modes and follow-up Q&A
• Professional, self-contained HTML reports with print-ready PDF output
• Markdown/JSON export and copyable report text
• Source lives in extension/ (MIT, like everything here)

cd extension
npm install
npm run build     # → extension/dist, load it unpacked via chrome://extensions

Or download the latest extension package, unzip it, open chrome://extensions, enable Developer mode, and choose Load unpacked. Version 0.2.0 is the first Chrome Web Store review package.

See extension/PUBLISHING.md for the release runbook and extension/STORE_LISTING.md for the Chrome Web Store listing. Known limits: DRM-protected players (Netflix etc.) can't be captured; live streams aren't supported; non-YouTube in-page videos are analyzed frames-only (no audio transcription).

Pro service architecture

The static site under site/ also contains email/password and configurable social sign-in with Supabase, plus Vercel Functions for one-time extension pairing, Stripe Checkout and Customer Portal sessions, signed subscription webhooks, atomic report quotas, managed OpenAI proxy calls, and opt-in report storage. See docs/AUTH_SIGN_IN.md for provider setup and account recovery. The database migration is under supabase/migrations/.

Required production environment variables are documented in site/.env.example. Private BYOK mode does not depend on any of these services and continues working if the Pro service is unavailable.

Roadmap

• Robustness: soft-fail transcription so frame-only videos still produce a report; clearer error surfaces in the UI
• More sources: webpage-rendered embedded video (Playwright), session-replay JSON exports (PostHog / Clarity / Hotjar / FullStory / LogRocket / OpenReplay), Zoom/Meet/Teams recordings
• Scene-change frame selection: smarter than the adaptive interval used in v0.1
• Embeddings + semantic search over processed timelines

Contributing

Issues and PRs welcome. The architecture is modular by design — most additions land in their own file under resolvers/, processors/, analysis/modes/, or outputs/ without touching the rest.

License

MIT — see LICENSE.