muvid

Tools to make music videos — four ways: from a song and a cover, from a pile of phone recordings of one gig, from an AI narrative pipeline, or as a lyric video whose words appear in time with the singing. See the table below for which is which; the rest of this section describes the AI pipeline.

The AI pipeline orchestrates the local ecosystem (falaw, lookbook, lacing, an, mixing) into a song-to-video pipeline. The user is the director; an agent (Claude in the terminal, or the local web UI) drives the stages.

Status (the AI pipeline): v0+. The pipeline (init → transcribe → align → cast → environments → script → render → compose) works end to end. Render strategies: lipsync, image_to_video, text_to_video, animation, still. CLI, Claude skill (.claude/skills/muvid/), and a single-page local UI all dispatch to the same Python functions. The v0 audit follow-up (improvement_ideas.md) shipped pluggable aligners, cost rollups + --budget, structured falaw progress events streamed to .muvid/fal_events.jsonl, end-to-end smoke fixture, lacing as the SSOT for word timings (no redundant whisper passes inside an), and a muvid.contracts adapter layer to sibling-package shapes. See misc/docs/design.md for the design rationale and misc/docs/alignment_references.md for the lyric-alignment literature muvid builds on.

muvid is four independent parts — the fourth is a plugin surface that currently ships three subgenres:

part	what it does	needs
muvid.visualize	a song + a cover → a 16:9 audio-reactive visualizer video, deterministic and ffmpeg-only	ffmpeg
the music_video genre	N phone recordings of ONE song → aligned, scored, cut and assembled into a music video	ffmpeg
the AI narrative pipeline (above)	a song → a cast, a script and generated shots	API keys, muvid[ai]
muvid.lyricvid	a song → a typographic music video: the words appear in time with the singing	ffmpeg, muvid[lyricvid]
muvid.choreo	a song and nothing else → event-driven visual music: onsets become objects, sections become scenes	ffmpeg
muvid.montage	a song + a pool of photos/clips → a montage cut on the beat, with Ken Burns, crossfades and grids	ffmpeg

Three of the four are free, deterministic and key-free, and are registered nw genres — so a host connector serves them directly. The AI narrative pipeline is the generative one. The lyric video is free by default; only its optional LLM creative director spends money.

muvid.lyricvid is also the first subgenre plugin — see Plugins for how to add another kind of video without editing muvid.

Install

pip install muvid                  # core: CLI + muvid.visualize + muvid.footage (needs ffmpeg)
pip install 'muvid[scoring]'       # footage scoring: quality + motion-to-beat + the weighted selector
pip install 'muvid[editor]'        # export a footage project as lacing annotations
pip install 'muvid[mcp]'           # serve the nw genres over MCP (fastmcp, py2mcp, nw)
pip install 'muvid[lyricvid]'      # lyric videos: the ASS renderer (pysubs2)
pip install 'muvid[lyricvid-web]'  # ...plus the browser renderer (playwright)
pip install 'muvid[ai]'            # the narrative pipeline (falaw, lacing, lookbook)
pip install 'muvid[ui]'            # FastAPI + uvicorn for the web UI

The narrative pipeline depends on local sibling packages (falaw, lookbook, lacing); with editable installs, install them first and use pip install -e ./muvid[ai]. muvid.visualize needs only mixing.

System: ffmpeg and ffprobe on PATH. Env (pipeline only): ELEVENLABS_API_KEY (transcription), FAL_KEY (fal.ai generation).

muvid.visualize — audio + cover → video

Turn a song into a publishable music video, without the AI pipeline:

from muvid.visualize import render_audio_video, list_visuals, verify_video, report

# Simplest: the cover on a 16:9 canvas, held for the song, loudness-normalized.
result = render_audio_video("song.wav", image="cover.png")

# Pick a visualizer (list_visuals() -> still, ken_burns, cqt, spectrum, waves,
# bars, scope), or pass "auto" / your own callable.
render_audio_video("song.wav", image="cover.png", visual="cqt", normalize=True)

# Check what you produced before shipping it.
print(report(verify_video(result.path, audio="song.wav")))

What it does, by default:

• 16:9, never pillarboxed — square/portrait art is composed onto 1080p filled with a blurred, darkened copy of itself, sharp cover centred on top.
• Loudness −14 LUFS (two-pass EBU R128), so a set of songs plays level.
• Video exactly as long as the song, H.264 High / yuv420p / AAC 48 kHz, +faststart, no edit lists — what YouTube asks for.
• A teal accent across the reactive visualizers (one tunable tint) over a muted background, so an album reads as one release whatever each cover's colours.
• A matching thumbnail derivable from the same composition (thumbnail_image, 1280×720, under YouTube's 2 MiB cap).

Every knob is overridable (visual, size, fps, normalize, CoverLayout(...), options={...}); add your own look with register_visual. Needs only ffmpeg — every built-in visual is ffmpeg-native except Ken Burns (via burns, which comes with mixing).

Publishing these to YouTube (single song or a whole folder as an album) is the yb package's job — it renders through muvid.visualize and uploads.

The music_video genre — footage assembly

Several people filmed the same gig on their phones. Each recording caught the same song through a different mic, from a different seat, starting at a different moment. muvid.footage puts them all on the clean studio master's timeline and cuts a music video out of them — no AI, no keys, no cost, just ffmpeg and arithmetic.

from muvid.footage import select_edl, validate_edl, derive_cuts
from muvid.footage.align import align_footage
from muvid.footage.edl import fill_gaps
from muvid.footage.assemble import assemble_music_video

song, dur = "master.wav", 205.0
clips = [("c1", "phone_a.mov"), ("c2", "phone_b.mp4"), ("c3", "tripod.mov")]

# 1. Where does each clip sit on the song? (audio cross-correlation, via mixing.audio)
aligns = align_footage(song, clips, song_duration=dur)

# 2. Who is on air over each span? A strategy turns alignments into an EDL,
#    fill_gaps makes it span the whole song, validate_edl is the one gate.
edl = validate_edl(
    fill_gaps(select_edl("best_confidence", aligns, dur), dur), aligns, dur
)

# 3. Cut it, over the clean master audio.
cuts = derive_cuts(edl, aligns, dict(clips))
assemble_music_video(cuts, song, "out.mp4", canvas=(1920, 1080))

The five stages

stage	module	what comes out
align	muvid/footage/align.py	per clip: offset_s, a confidence, its coverage of the song, overlaps
score	muvid/footage/scoring/	a tensor S[clip, frame, metric] on ONE shared song-time grid (hop≈0.1 s)
select	muvid/footage/strategy.py, select_score.py	an EDL — which clip covers which span
validate	muvid/footage/edl.py	the one gate: ordered, non-overlapping, gapless, inside each clip's coverage
assemble	muvid/footage/assemble.py	one ffmpeg pass per cut → concat by stream copy → mux the master

What the design commits to

• Nothing vanishes because it measured badly. A clip that doesn't overlap the song is still recorded, still listed, still addressable — it just carries overlaps=False.
• A hole is an explicit gap entry, not an absence. EdlEntry(clip_id="") (null over JSON) means "no footage here, fill it", so the render is always exactly the song's length and every span is accounted for by exactly one entry.
• Bounded memory. One ffmpeg invocation per cut, then a stream-copy concat — peak memory is O(1) in the cut count, not O(n). A 70-cut score-driven edit renders on the same box as a 3-cut one.
• Mixed phones are the normal case. Clips are scaled and padded onto a fixed canvas (landscape / portrait / square), rotation metadata is honoured, and mismatched frame rates are unified — so a portrait iPhone clip and a 29.97 fps camera cut together without drift.
• The strategy is a config object, not a branch. best_confidence (default), longest_take and fewest_cuts work from the alignments alone; weighted runs a beat-snapped Viterbi over the score tensor. Add your own with register_selection_strategy.

Scoring (pip install 'muvid[scoring]') resolves every clip's every metric onto that one song-time grid, so the same numbers drive both the automatic cut and a human editor's lanes. The core tier — sharpness, exposure, shake, face framing, motion-to-beat, the selector — is torch-free and MIT/BSD/Apache/ISC only, and is what a deployment installs. The lip-sync tier (muvid[scoring-lipsync]: Demucs + SyncNet) is opt-in and off by default: the htdemucs weights are CC-BY-NC (research-only, not commercial-clean), and it peaks at 2–3 GB on CPU. Turning it on takes the extra, MUVID_SCORING_ENABLE_LIPSYNC=1, and weights you point at yourself via MUVID_SYNCNET_S3FD_WEIGHTS / MUVID_SYNCNET_WEIGHTS — muvid never downloads model weights at runtime. Without all three it skips cleanly rather than scoring zero.

As a hosted genre. muvid.genre_music_video registers music_video as an nw.Genre (canvas presets as its Templates) with a project factory. Every operation on a footage project — set the song, add or remove a video, find where each fits (align) or place one by hand (set_offset), score, cut (propose_edit), keep and change named edits cut by cut (save_edit, set_cut, split_cut, merge_cut, replace_edit), trim an edit to part of the song (set_span), render — is a plain function in muvid.footage.service, listed once in its FOOTAGE_OP_SPECS and registered with nw as nw.GenreOps, so a host serves them without importing muvid's internals. muvid.mcp.footage_tools and muvid.mcp.scoring_tools expose the same operations as MCP tools (set_song, add_footage, align_footage, propose_edit, assemble_music_video, footage_set_cut, footage_render, …) over a stateful per-user workspace under ~/.local/share/muvid (MUVID_DATA_HOME to relocate). A host that serves the project itself (the reelee studio) creates it where it wants with projects_dir — a muvid.Project, an nw.Project whose media is registered in the host's artifact catalog — and python -m muvid.importing MANIFEST PROJECTS_DIR brings a finished production into one.

As an editor document. pip install 'muvid[editor]' adds muvid.footage.lacing_bridge, which exports a project as lacing standoff annotations — three body schemas, clip-alignment/v1, clip-score-track/v1 and music-video-edl/v1, all in song time on one axis — and reads an edited DECISION tier back into an EDL. Annotate → edit → export → render round-trips to the same cuts, transitions included.

muvid.lyricvid — lyric videos

A song in, a typographic music video out: the words appear in time with the singing. Free, deterministic, and no API key.

pip install 'muvid[lyricvid]'
python -m muvid.lyricvid render song.wav out.mp4 --lyrics lyrics.md

That is the whole minimum. Everything else is optional: --lyrics may be muvid's lyrics markdown, plain text, an .srt, an enhanced .lrc, an existing muvid project — or nothing at all, in which case the words are transcribed.

The pipeline is four seams, each with a default that genuinely works:

audio (+ lyrics)  ->  TimedText     measured word times, from muvid's own aligner
                  ->  TreatmentSpec the creative decision
                  ->  Scene         every position and time, computed in Python
                  ->  mp4           the ASS renderer, or the browser one

The layout is chosen from a closed set of archetypes — one word centred, stacked lines, a karaoke wipe, a fixed concrete page whose words ignite in reading order, words packed into a shape, text on a path, scatter:

python -m muvid.lyricvid vocabulary          # the archetypes, with descriptions
python -m muvid.lyricvid analyze song.wav    # words/second, sections, timing quality
python -m muvid.lyricvid propose song.wav    # 3 ranked treatments, free, no model

A treatment is a small JSON document with two layers: a direction (mood, palette, typography, motion vocabulary, and a rationale) and scenes (an archetype plus parameters). It never contains a coordinate or a timestamp — those are computed from measurement, which is what keeps the words in sync. Edit it by hand, validate it, re-render:

python -m muvid.lyricvid validate treatment.json
python -m muvid.lyricvid render song.wav out.mp4 --treatment treatment.json

An LLM creative director can propose the treatment instead (--use-llm). It is opt-in, it is the only part of muvid that spends money, and it still emits only a treatment — never rendering code, never coordinates.

Two renderers. ass (default) writes an ASS subtitle file and burns it in with ffmpeg: frame-exact, no browser, and the .ass file is itself an editable deliverable you can open in Aegisub. web drives headless Chromium for effects ASS cannot express, at 10-100x the render time.

muvid.choreo — visual music

Audio in, nothing else. The Oskar Fischinger / Norman McLaren / "Star Guitar" lineage: discrete musical events become objects with persistence on screen, and sections become different arrangements — choreographed and structural, unlike the continuous spectrum readout of muvid.visualize.

python -m muvid.choreo render song.wav out.mp4 --archetype star_guitar

Four archetypes: fischinger (shapes ignite on a grid per onset), star_guitar (a scrolling landscape — poles on the kicks, buildings on the mids, and their spacing is the rhythm), mclaren (scratches on black), swarm. Analysis is numpy only — a three-band spectral-flux onset detector with hysteresis and a beat-derived minimum inter-onset interval, a beat grid (from mixing when its librosa is present, else a built-in estimator), sections from energy. Frames are drawn in numpy and piped into ffmpeg. Deterministic: same song and seed, same video. events.json and scene.json ride along as inspectable artifacts.

muvid.montage — beat-cut montage

A song plus a pool of photos and/or short clips that have no timeline of their own — the thing "photo beat sync" templates sell.

python -m muvid.montage render song.wav out.mp4 --photos a.jpg b.jpg c.jpg
python -m muvid.montage plan   song.wav --photos a.jpg b.jpg c.jpg   # the EDL, as JSON

Its core is a planner: a pool with no timeline → slots on the song's beat grid and section structure, with section-aware pacing (choruses cut twice as fast, intros half as fast, never a hold over ten seconds) and an explicit reuse policy so twelve photos can carry a three-minute song — never the same image within n cuts, each revisit a different crop, the strongest images reserved to lead the final chorus, a cover pinned to the first and last slot. Four archetypes: ballad_dissolve, beat_cut, grid (2×2 tiles swapping on the beat), stop_motion. Rendering is one ffmpeg graph. plan.json is written before rendering and rides along as an artifact, so an edit is a re-render.

Plugins — adding a kind of video

muvid.subgenres is how another kind of video gets added, by muvid or by anyone else, without editing muvid. A subgenre is declared as pure data:

from muvid.subgenres import Subgenre

RANSOM_NOTE = Subgenre(
    slug="ransom-note",
    title="Ransom note",
    description="Lyrics cut from magazines, one word per beat.",
    render="my_package.render:render",  # a STRING, imported lazily
    inputs={
        "type": "object",
        "required": ["audio"],
        "properties": {"audio": {"type": "string"}},
    },
)

and shipped by pointing an entry point at the manifest:

[project.entry-points."muvid.subgenres.v1"]
ransom-note = "my_package.subgenre:RANSOM_NOTE"

The renderer is a plain function, (RenderRequest) -> RenderResult, handed paths and primitives — never a muvid project object, so muvid can change its own schema without breaking you.

Two properties are worth the design they cost. Listing imports nothing: the entry point resolves to a manifest, not a renderer, so a UI or an agent can choose among a dozen installed subgenres in microseconds, and a plugin whose heavy dependency is missing becomes a recorded error rather than a dead catalogue. It composes with nw rather than competing: muvid bridges subgenres up into the nw genre catalogue, so a plugin depends on muvid alone and a host connector picks it up for free.

Verify yours against the contract:

from muvid.subgenres.testing import check_subgenre_conformance

report = check_subgenre_conformance(RANSOM_NOTE, workdir=tmp_path)
assert report.ok, report.summary()

python -m muvid.lyricvid subgenres    # everything installed, as JSON

choreo and montage are themselves registered this way and nothing else — nothing in muvid names them. They appear in the nw genre catalogue through muvid/genre_subgenres.py and are renderable over MCP through the generic render_subgenre tool, which fetches every file input into the caller's workspace and validates the request against the manifest's own schema first.

30-second tour

Every verb takes the project root explicitly. Ask muvid where a project belongs rather than inventing a path: an app-named folder under your home directory is data sitting where only code belongs, and nothing backs it up or cleans it up.

root="$(muvid project-root park-bench)"   # → ~/.local/share/muvid/projects/park-bench
                                          # $MUVID_DATA_HOME relocates the root

# Bootstrap a project around a song.
muvid init "$root" --song ~/Downloads/park_bench.mp3 --title "Park Bench"

# Transcribe to a draft lyrics.md (you'll edit it).
muvid transcribe "$root"

# … you edit lyrics/lyrics.md to fix mishears and add [section] tags …

# Align lyrics.md against the transcript and write lyrics/alignment.annot.
muvid align "$root"

# Cast a character: card, then images, then lookbook curation.
muvid character "$root" maya --description "mid-30s, dark curly hair, wary eyes"
muvid character-generate "$root" maya --n 6
muvid character-curate    "$root" maya --k 8

# Establish an environment.
muvid environment "$root" park_bench --description "wooden park bench at dusk"
muvid environment-render "$root" park_bench

# Write/edit script/script.md (let an agent draft it from the lyrics + cast),
# then sync it back into project.json:
muvid script-apply "$root"

# Estimate cost before committing fal calls.
muvid estimate-cost "$root"

# Render every shot (optionally gated on a USD budget), then composite.
muvid render  "$root" --budget=2.50
muvid compose "$root"
# → "$root"/output/final.mp4

# Inspect progress.
muvid status        "$root"           # human-readable
muvid status --json "$root"           # structured shape

# Or open the local UI (FastAPI + single HTML page).
muvid serve "$root"

Pluggable aligners

muvid align --aligner=... accepts:

• scribe-greedy (default) — Scribe transcript + greedy token-match.
• user — caller-supplied line_index → (start, end) timings.
• whisperx-lite — local faster-whisper, falls back to scribe-greedy if no audio_path= is given.
• stars — singing-grade joint inference (stub; NotImplementedError).

Plug your own with muvid.align.register_aligner(name, fn, ...).

Interactive character curation

When a recipe's automatic top-k isn't quite right, replay a JSON of decisions:

# decisions.json:
# [{"keep": ["<image_id>"], "reject": [...], "stop": false}, ...]
muvid character-curate-interactive "$root" maya \
    --decisions decisions.json --k 8 --present 6

How it fits the ecosystem

Concern	Owner
AI media (TTS, image, video, lipsync, voice clone)	falaw
Reference image curation (LoRA-style sets)	lookbook
Timeline / interval annotations (lyrics, sections)	lacing
Structured 2D animation (cutout characters)	an
Audio/video editing, alignment + ElevenLabs Scribe	mixing
Genre / Template registry, durable async jobs	nw
Visualizer, footage assembly, pipeline, dispatcher	muvid

muvid is the orchestrator: a folder layout (project.json + song/, lyrics/, characters/, environments/, script/, shots/, output/), a content-addressed cache (re-render only what changed), and a uniform dispatch layer with three surfaces (CLI, skill, UI) all calling the same Python functions in muvid.facade.

Render strategies

Each shot picks one. The dispatcher resolves shared inputs (audio slice, lyric lines that fall in the shot interval, character / env anchor images) once and hands them to the strategy:

strategy	use it for	calls
lipsync	character singing on screen	falaw.animate_face
image_to_video	cinematic shot, env anchor as i2v seed	falaw.image_to_video
text_to_video	no anchor, pure prompt	falaw.text_to_video
animation	stylized 2D cutout	an.orchestrate
still	single image held for the duration	ffmpeg

The Claude skill

.claude/skills/muvid/SKILL.md walks Claude (or any agent that follows Claude Code skills) through the eight stages. It will:

• run muvid status first to see where you are
• pick the next stage and offer to run it
• never re-transcribe after you've edited lyrics.md
• never --force a render without asking
• offer to draft script/script.md from your lyrics + cast

Layout

muvid/
  __init__.py         public surface (the facade)
  __main__.py         CLI (cw)
  schema.py           ProjectSpec, ShotSpec, SectionSpec, …
  project.py          MusicVideoProject (folder facade)
  lyrics.py           transcribe + parse/render lyrics.md
  align.py            pluggable aligners + lacing SqliteStore writer
  characters.py       cards + ref images + lookbook curation (incl. interactive)
  environments.py     cards + establishing-image generation
  script.py           script.md ↔ ShotSpec list
  cost.py             render-cost rollup over pending shots
  events.py           pipe falaw progress events → .muvid/fal_events.jsonl
  contracts.py        adapters: muvid SSOT ↔ falaw / an / lacing shapes
  renderers/
    __init__.py       dispatcher + RenderContext + caching
    lipsync.py        falaw.animate_face
    image_to_video.py falaw.image_to_video
    text_to_video.py  falaw.text_to_video
    still.py          ffmpeg single-image loop
    animation.py      handoff to `an.orchestrate` with lacing-driven lipsync
  compose.py          ffmpeg concat + overlay song audio
  facade.py           top-level verbs the CLI/skill/UI call
  downloads.py        claim()/resolve() — muvid owns resolution, the host owns transport
  visualize/          part 1: the ffmpeg-only audio visualizer (+ its visual registry)
  footage/            part 2: align, edl, strategy, select_score, assemble, workspace
    service.py        the footage operations (the SSOT every surface derives from)
    scoring/          the score tensor: grid, frames, quality, motionbeat, segment, lipsync
    lacing_bridge.py  project ⇄ lacing standoff annotations (the editor document)
  genre.py            registers the `music_visualizer` nw genre (and imports the next)
  genre_music_video.py  registers the `music_video` footage genre and its operations
  production.py       muvid.Project — a production a host places and serves
  catalog.py          registers hosted media in the host's artifact catalog
  importing/          python -m muvid.importing — finished productions into a host
  mcp/                the MCP tool surface: tools, footage_tools, scoring_tools
  ui/
    app.py            FastAPI app
    static/index.html single-page UI
.claude/CLAUDE.md               agent & contributor guide (start here)
.claude/skills/                 muvid, muvid-visualize, muvid-score-footage,
                                muvid-choose-footage-segments
misc/docs/design.md             part 3's design rationale
misc/docs/footage_scoring_design.md    part 2's LOCKED design decisions
misc/docs/footage_scoring_research.md  the citations behind them
misc/docs/improvement_ideas.md  v0 audit + post-audit follow-through