Life2Film Director
A personal agent that turns a day of your own footage into a trailer, and learns your taste from every
correction you make.
Built solo in one day at the Build Personal Agents Hack (Neon, Terra Gallery, San Francisco, 4 Oct 2026).
MIT licensed. Status: hackathon prototype, works end to end on the author's Mac; the hosted demo covers
memory, search and scripts.
Hosted demo: tech-week-agent.vercel.app · Video (2:51): youtu.be/NAmEDA6vIEY · Write-up: rustman.org
What it does
- Knows your footage. Clips from a DJI pocket camera and phones, with speech (whisper, word
timings) and picture quality (31 measurements per frame, shot boundaries) in Neon Postgres:
days → clips → sentences / moments. Old days come from the video-analyzer sidecars; new clips
(shot today, or mailed to the agent's AgentMail inbox from a phone) are scored in Node by the
life2film-engine WASM core, transcribed, and
captioned by a vision model.
- Remembers like a family. Every sentence and every clip caption has an embedding
(
lakebase_vector, cosine ANN) and a Russian+English tsvector (lakebase_bm25); search_footage
fuses both with RRF. "Найди, где дочка говорит про дом" returns the clip, the second and the line;
"закат у океана" finds the picture even when nobody spoke.
- Writes the script. "Собери трейлер недели в долине, 3 минуты" → the director reads the whole
week as text (2,367 sentences across 163 clips) and your standing rules, and writes
script.json:
a cold open, one hook line, three acts, speech kept as one track, cutaways that show what is being
said, drives with their own sound.
- Renders it.
vlog_cut.py lays the script out as OpenTimelineIO, the Rust renderer cuts the mp4
on the Mac (on Fly.io this is a sprite job). The render runs in the background; the chat card shows
the renderer's progress lines and turns into a player with a poster when the file is ready.
- Learns. "Первые 10 секунд скучные, нужен хук" → the remark becomes a durable rule
(
rules table), the script is revised, the opening re-cut. Ten rules were seeded from a month of
real corrections; every new one is shown as a card.
Screens
| Live analysis of an uploaded clip | Search card | Renders and rules |
|---|
| | |
Live: https://tech-week-agent.vercel.app — chat, search, scripts, rules, and clip analysis in your browser
(the WASM engine scores frames on-device; only scores and one frame are sent, nothing is uploaded).
Speech, sidecars and renders need the Mac build. The hosted demo is bring your own OpenAI key: paste it in the
header, it stays in your browser and travels only as a request header; no server-side key is configured.
Demo video (3 min): https://youtu.be/NAmEDA6vIEY
Stack (hack sponsors, each doing a real job)
| |
|---|
| Neon Postgres | footage index, scripts, renders, the rules (taste graph), Mastra memory |
| Neon AI Gateway | LLM path when enabled; falls back to Anthropic / OpenAI (src/lib/llm.ts) |
| Mastra | the director agent and its tools (src/mastra) |
| assistant-ui | chat with generative cards: days, script timeline, player, rule learned |
| AgentMail | the agent's own inbox: clips sent from a phone become footage (check_inbox) |
| Exa | event / venue facts for captions and descriptions (next) |
| Kernel | posting the reel through a real browser (next) |
| Fly.io | hosting; renders as sprites (next) |
Use it from your own agent (skill)
skills/life2film-director/SKILL.md is an agent skill: install with npx skills add fortunto2/tech-week-agent
(or point your agent at the file) and it gets the five install commands, the tool table and the failure
messages. The tools in src/mastra/tools are plain Mastra createTools and work in any Mastra agent.
Run
pnpm install
cp .env.example .env.local # Neon DATABASE_URL, an LLM key, FOOTAGE_ROOT, VIDEO_ANALYZER_DIR/BIN
pnpm db:push # schema → Neon
pnpm seed:rules # the owner's editing rules
pnpm ingest ~/Movies/trip/day1 "Day 1" -5 # video-analyzer sidecars → Neon (tz offset for filename times)
pnpm ingest-clip ~/Movies/today/clip.MP4 # a clip with no sidecars: engine + whisper + caption
pnpm embed && pnpm caption # embeddings for sentences; VLM captions per clip
pnpm dev # http://localhost:3000
The analysis sidecars (*.va.stt.json, *.MP4.va.otio) and the renderer come from the author's
open video-analyzer pipeline; the agent, schema, tools and UI were written during the hack.
Layout
src/db/schema.ts days, clips, sentences, moments, rules, scripts, renders, feedback
src/lib/sidecars.ts readers for the analysis artefacts
src/lib/day-text.ts the day as numbered text (what the director reads)
src/lib/script-schema.ts script.json contract (+ validation against the footage)
src/mastra/agents/director.ts
src/lib/analyze.ts life2film-engine (WASM) frame scoring + shot detection in Node
src/lib/search.ts Lakebase hybrid search (vector + BM25, RRF) and picture search
src/lib/ingest-clip.ts new clip → engine, whisper, caption, embeddings → Neon
src/mastra/tools/ list_days · search_footage · write_script · get_script · render_script · learn_rule · list_rules · list_renders · check_inbox
src/app/api/chat/route.ts Mastra → AI SDK v7 stream → assistant-ui
src/app/tool-ui.tsx generative cards per tool
scripts/ingest.ts folder → Neon
