Souffleur

A voice interface for thinking, with a repertoire that grows through use.

Capture happens on a phone while driving; the Workspace is a responsive web app for seeing what was recorded and what was derived from it. See Notes.md for the research framing.

---

Quick start

cp .env.example .env          # then fill in GitHub OAuth + LiteLLM values
docker compose up -d          # Postgres only (pg-boss lives in the same DB)
pnpm install
pnpm db:migrate
pnpm dev                      # web on :3000, worker alongside it

Open http://localhost:3000 and hit Start recording. No account, no OAuth app, nothing to register — you get a guest identity and the starter repertoire (mark, diary, to-doc, interview, …) immediately.

Then, to get something on screen without driving anywhere:

pnpm db:fixtures       # a demo session: 12 chunks, transcript, synthesised audio

This writes chunks already marked transcribed, so the Workspace, the transcript view and provenance seeking all work without a LiteLLM key and without a microphone.

Accounts: guest by default

A guest is a real user row behind a cookie, so sessions, repertoire and invocations are scoped exactly as for a signed-in user and the growth curve is measured identically. Nothing about the research data is second-class.

But guest identity is cookie-scoped. Clear site data, switch browsers, or record from a second device and you are a different person as far as the database is concerned — your commutes split across two accounts and the growth curve fragments. For the longitudinal deployment the paper depends on, sign in properly on day one.

Signing in migrates everything the guest recorded onto the real account (packages/db/src/link-guest.ts). This matters more than it looks: Better Auth deletes the guest user on link, and every domain table cascades from user, so the migration has to move the data first or signing in would destroy the recordings. That path is integration-tested — see link-guest.test.ts.

GitHub OAuth is optional. Leave GITHUB_CLIENT_ID / GITHUB_CLIENT_SECRET empty and the sign-in button simply does not appear. To enable it, register an app at https://github.com/settings/developers with callback http://localhost:3000/api/auth/callback/github. Register a second app for the deployed domain; one app cannot serve both.

Wake lock is what makes this work. iOS Safari suspends audio capture the moment the screen locks or the browser backgrounds. Mount the phone, plug it in, keep /record in the foreground. The recorder requests a screen wake lock and warns you when it could not get one.

---

Testing on a real phone

getUserMedia requires a secure context. localhost is exempt, but a phone on your LAN is not — so use a tunnel:

pnpm tunnel                   # cloudflared → https://<random>.trycloudflare.com

Add that URL as the callback on a GitHub OAuth app and set BETTER_AUTH_URL to match, otherwise sign-in will redirect to the wrong host.

---

Architecture

apps/web      Next.js — Workspace UI, recorder PWA, API routes, Better Auth
apps/worker   pg-boss worker: transcription, classification, extraction, macros
apps/pipecat  Python voice agent (see TALKBACK.md)
packages/db   Drizzle schema + migrations   ← shared contract
packages/llm  LiteLLM wrappers (chat, transcribe)
packages/shared  Zod contracts, offset arithmetic, the directive gate  ← shared contract
packages/talkback  What the agent knows: retrieval, prompt, setting profiles
packages/workspace  Pure fold/extract/classify/macros/trajectory logic

Packages ship TypeScript source rather than build artefacts. Next transpiles them; the worker runs them through tsx. There is no per-package build step.

Two decisions worth knowing before you change things

The web app never enqueues jobs. It only writes rows. The worker sweeps for chunks in stored and queues them itself. A worker outage therefore cannot lose work — chunks accumulate and drain on the next start — and the burst that arrives when a phone regains signal after a tunnel is absorbed naturally.

Every chunk is an independently decodable audio file. The obvious approach, MediaRecorder.start(timeslice), only puts the container header in the first blob; later blobs are bare fragments no decoder will read alone. Since chunks are transcribed independently, that silently breaks the pipeline. The recorder recycles the MediaRecorder per chunk against a persistent MediaStream instead. The cost is a few milliseconds of audio at each boundary, which is why findCoverageGaps tolerates sub-250ms discontinuities.

The capture path

MediaRecorder → IndexedDB queue → POST /chunks → filesystem + audio_chunk row
                                                        ↓
                        worker sweep → pg-boss → LiteLLM Whisper → utterance rows
                                                        ↓
        ┌───────────────────────────────────────────────┴────────────────────┐
        │                                                                    │
  classify.utterance (per chunk, ~20s)                  workspace.extract (per 8, ~60s)
        │                                                                    │
  directive rows ──→ invocation ──→ macro_proposal              workspace_op rows
        │                                                                    │
        └────────────────────→ /api/record/cues ←────────────────────────────┘
                                (the secondary display)

Both lanes feed the cue panel on /record, and both come out of Postgres — never from the voice container. That is the point: kill Pipecat mid-recording and the panel keeps filling, because it was never downstream of the conversation.

Chunks are written to IndexedDB first and deleted only once the server acknowledges them. A commute goes through tunnels; without the local queue, losing signal loses the recording.

Stop does not end the recording — it opens the debrief. The three questions from /study go on screen and the same chunk loop keeps running underneath, up to ninety seconds or until Done, marked on the session by debrief_started_offset_ms / debrief_ended_offset_ms. That interval is the one part of a drive researchers may read (/study promises nobody reads the rest), so the boundary is a property of an interval rather than of a session. Talk-back stops at Stop; capture does not. See PILOT_01.md for why this exists: the first pilot's most useful material came after ended_at and survives only on somebody's camera.

Audio is transient. A chunk survives only until its transcript is committed, then the file is deleted and storage_key is nulled. The transcript is the record; the audio is scaffolding. It cannot be dropped at upload time, because audio has to outlive a LiteLLM outage — the recorder discards its own copy as soon as the server acks.

Set KEEP_AUDIO=true to retain it. Do that before changing CHUNK_MS or MODEL_TRANSCRIBE: once audio is gone the corpus cannot be re-derived, and the transcript is all you will ever be able to analyse.

---

Data model

The schema is the paper's measurement apparatus, and the repertoire growth curve cannot be reconstructed after the fact. Three consequences:

• utterance.text is append-only and never mutated. This is the asymmetry that makes misclassification survivable: artefacts are derived, so an error blemishes but never destroys.

  The one write after insert is kind, and it is deliberately narrow. The classifier fills it where kind = 'unclassified' — a monotone one-way fill of a derived default, so running twice cannot change an answer and a failed job simply leaves the row for the next sweep. text is never touched and human corrections still go to kindOverride, which is the only column the Workspace writes.

• capabilityVersion is append-only, so edits are measurable, not just creations. capabilityOrigin records createdVia and the triggering session.

• invocation records every fire — including rejected and reverted ones. That is where frequency-ordering and "which capabilities survived" come from.

artifact.spans carries provenance back to source utterances. Treat it as a constraint, not a feature to add later.

• directive is one row per direction, keyed by the utterance that carried it, with the verb, the object and a restatement to read back. A capabilityId of NULL is the interesting case, not the failure case: it is an operation the person invented, and those are exactly what the macro detector mines.
• macro_proposal holds crystallisation candidates, including declined ones. A declined proposal is never re-offered, and "what they tried to add and failed" is a stated field-study measure that only survives if refusals do.
• capture_session.setting is fixed once, at session creation, and never updated — not even when a session is resumed. It governs turn-taking and how much goes on screen, so a recording whose halves ran under different rules would not be interpretable under either. It is detected, not asked (apps/web/src/lib/recorder/detect-setting.ts): a laptop is desk; on a phone the accelerometer separates walking (periodic steps), driving (aperiodic vibration), and a still phone lying flat (desk) or propped up (hands_busy). With no motion data the default is driving. The recorder shows what it inferred and offers a correction; recording_started carries setting_source so the correction rate is measurable.

Where the setting is read

One value, two consumers, so the voice and the screen can never disagree about whether the person can look at anything:

Consumer	Reads	Effect
composeSystemPrompt	SETTING_PROFILES[s].stanza, maxReplyWords	reply length, and whether the agent may mention the screen
/api/record/cues + CuePanel	displayAllowed, density, maxContentCues, maxDirectionCues	whether the panel exists, how it renders, and how much is on it

displayAllowed and density are separate questions on purpose. The first asks whether a screen exists; the second asks what to do with it. Collapsing them is how the panel came to render its at-110km/h design — five rows, eight words, an eight-second dwell floor — for somebody sitting at a desk. driving and walking render no panel at all, so those rules had never once run in the setting they were written for.

Density	Setting	Reads as
glance	hands_busy	flat, cut to eight words, held still for eight seconds
read	desk	grouped by topic, typed by block kind, nothing truncated

driving reproduces the stance the base prompt was written with, so an unset setting and an explicit driving behave identically — which is what makes the column safely nullable for every recording made before it existed.

packages/talkback/src/setting.ts is exported as @voicemural/talkback/setting as well as from the index. The recorder is a client component and must reach the profiles through that subpath: the package index re-exports retrieval.ts, which imports @voicemural/db, and importing it from the browser drags the Postgres driver into the client bundle and fails the build.

---

Commands

Command	Effect
pnpm dev	web + worker with hot reload
pnpm typecheck	all packages
pnpm test	unit tests (offset arithmetic especially)
pnpm db:generate	generate a migration from schema changes
pnpm db:migrate	apply sql/init.sql (extensions, pgboss schema), then migrations
pnpm db:studio	Drizzle Studio
pnpm db:seed	re-install starter repertoire for all users
pnpm db:fixtures	seed a demo session with synthesised audio + transcript
pnpm icons	redraw the favicon, apple-touch and PWA icons from scripts/generate-icons.mjs

pnpm db:fixtures seeds three sessions rather than one, with the workspace ops and the macro proposal they would have produced. Both are written directly: extraction and induction need a model, and the whole point of the fixture is that /workspace, /trajectory, /repertoire and /board all render without a LiteLLM key. It also picks hands_busy for the most recent session, so the cue panel is visible on /record without a car, and it enables the board for the seeded user — with one task speech moved to done and the person having moved it back, so the reversal marker is visible without anyone clicking.

The task board is hidden per participant until user.board_enabled_at is set. There is no admin UI for it; enable someone with pnpm db:studio or UPDATE "user" SET board_enabled_at = now() WHERE id = '…';. pnpm workspace:show prints tasks as (task/<state>).

Starting with a board you already have. Nobody arrives with nothing in flight, and the extractor only ever hears a task that gets said — so until the work someone is already carrying is on the board, the first drives talk past it. /board/import takes a Trello JSON export, a Jira or Trello CSV, a Notion table, or a plain list, and adds the tasks as cards. The columns other tools use are mapped onto the five tenses (taskStateFor in packages/workspace/src/board-import.ts); a heading the mapping does not recognise becomes a topic rather than a column, which is what lets a Trello-shaped export and a Notion-shaped one both read correctly.

Three things about it are deliberate, and all three are load-bearing:

• The parse happens in the browser. A pasted export carries assignees, reporters and whatever else the source adds; only the lines the person confirms are sent. parseBoard is pure, so it runs where the paste already is.
• Imported ops carry via: "import", and judge() does not score them. They are the person's own record of their own work, not a reading of their speech; a hundred imported cards sitting untouched would otherwise read as a hundred accepted machine-made transitions. Everything speech does to them afterwards is judged as usual — which is the point, because it gets the acceptance question asked on the first drive instead of the tenth.
• Imported blocks carry no spans. No utterance said them, so there is nothing to seek back to, and the card says where it came from instead.

loadUserOps keeps import ops alongside the manual and agent ones, so workspace:rebuild restores an imported board rather than emptying it.

Study participants and conditions have validating commands rather than SQL, because a mistyped condition does not fail — the drive silently runs the defaults:

Command	Does
pnpm study:participant --user <id> --id P07	the pseudonym PostHog and the export join on (--clear removes it)
pnpm study:condition --user <id> --set '{"agendaOffers":true}'	the condition for that person's NEXT drives; each drive freezes its own copy (--clear resets to today's behaviour)
pnpm study:condition --user <id>	print the template and the last ten drives' conditions
pnpm study:export	one JSONL file per participant into storage/study-export/ — counts, timings, ids and enums, no transcript, agent or workspace text
pnpm study:metrics [--user <id>] [--print]	the four measure groups — system behaviour, steerability, thinking, relief — per drive and per participant, into storage/study-metrics/
pnpm study:review --user <id>	the cards awaiting a day-7 verdict, oldest first
pnpm study:review --user <id> --card <id> --outcome done|open|lost	record one verdict

--include-text on the export is refused unless --user is listed in STUDY_PILOT_USER_IDS: it exists for the researcher's own pilot account only.

study:metrics reads transcript text to classify corrections and repeat requests, and returns counts only — the classification happens inside the privacy boundary and no text reaches the file. --re-prompt-ms sets how long the agent's silence must last before the participant's next words count as a re-prompt (default 5000); --tz sets the zone whose calendar days the revisit measures count in, overriding STUDY_TIME_ZONE for that run.

---

Deployment (Coolify)

Postgres is a separate Coolify database resource, not part of the compose file. Coolify's scheduled backups only cover managed database resources, and this database is the study's measurement apparatus — the growth curve cannot be rebuilt after the fact, so an unbacked-up container volume is the largest risk in the whole deployment.

1. Create a PGVector (17) database resource. Match the major version to docker-compose.yml, so pg_dump/restore between prod and a laptop stays a non-event. Enable scheduled backups now, not later.
2. Add the repo as a Docker Compose resource, set the compose file to docker-compose.prod.yml, and connect via the GitHub App — every push to main then redeploys with no webhook config. Traefik terminates TLS and supplies the public HTTPS domain, which the recorder needs anyway.
3. Enable Connect To Predefined Network on that resource, or the services cannot resolve the database by its internal hostname.
4. Point a DNS A record at the Coolify host and set the domain on the web service. Then set BETTER_AUTH_URL to that origin — https://voice.example.com, no port, no trailing slash. It is set by hand rather than derived from Coolify's SERVICE_FQDN_WEB_3000, which renders as https://host:3000; Better Auth builds its OAuth callback and validates origins from this value, so a port browsers never use breaks sign-in in a way that looks like GitHub's fault.
5. Set the rest in Coolify's environment: DATABASE_URL (the database's internal URL), BETTER_AUTH_SECRET, GITHUB_CLIENT_ID, GITHUB_CLIENT_SECRET, LITELLM_BASE_URL, LITELLM_API_KEY, and the three MODEL_* roles. Optionally KEEP_AUDIO=true.
6. Register a second GitHub OAuth app for the deployed domain, callback https://<domain>/api/auth/callback/github — it must match BETTER_AUTH_URL exactly. One app cannot serve both localhost and prod.

The migrate service runs to completion before web and worker start. It applies packages/db/sql/init.sql — extensions and the pgboss schema — before the migrations, so a fresh database needs no manual setup step. Every statement there is IF NOT EXISTS and it runs on every deploy.

Read the worker's log after the first deploy: it reports on startup whether LiteLLM is reachable and whether MODEL_TRANSCRIBE exists on that instance. Getting either wrong is otherwise invisible — chunks just accumulate as stored.

---

Working in parallel

packages/db and packages/shared are the contracts — changing either is a two-person decision. Otherwise the seams are independent:

Seam	Scope
A — Capture	recorder PWA, IndexedDB queue, chunk upload, storage, the cue panel
B — Pipeline	worker jobs, transcription, classification, invocation, macros
C — Workspace	workspace/trajectory/repertoire UI, growth curve, exports

A and B meet only at the utterance table. C consumes both and touches neither. The cue panel sits in A but reads B's output over HTTP, so it never reaches into the pipeline's tables directly — packages/db/src/display.ts is that seam.