Lecture Copilot

Ask a question about a lecture video and get an answer where every sentence cites the moment it comes from, like L4 · 31:32. Clicking the citation moves the video to that second.

Live demo: https://lecture-copilot.monishpatalay.dev (two courses, five lectures; no sign-in needed to ask)

In the screenshot the answer cites L4 · 31:32, and the transcript line at 31:32 is where the lecturer says it ("maybe make it a dynamic array or a linked list").

What it does

For students	For professors
Ask a lecture or a whole course. Answers stream in, cite their sources and say "Not covered in these lectures" when the lectures don't answer the question. Questions about the course itself ("how many lectures are there?") are answered from its outline.	Upload a video in the browser (MP4, MOV or WebM, up to 2 GB and 2 hours). It is transcribed, chaptered and indexed without anyone's laptop being on, and an email says when it is ready.
Follow up ("why?", "explain that more simply") and the earlier exchange is taken into account.	Insights: what students ask, which questions the lectures didn't cover, which answers got a thumbs down, and the most-cited moments.
Exam prep: multiple-choice questions written from each lecture, scored, each linking to the moment that explains the answer. A wrong-looking question can be reported.	Exam prep editing: correct or delete individual questions, see which ones students reported, or have a new set written.
Rate answers with a thumbs up or down.	Manage: rename or delete courses and lectures. Uploading is invite-only: people request access and the admin approves.

There is also a Help page with answers to common questions and a contact form that emails the site's owner.

The screenshots predate the Help link in the sidebar.

How it works

flowchart LR
  subgraph Upload
    B[Browser] -- "presigned PUT" --> R2[(Cloudflare R2)]
    B -- "queued" --> PG[(Postgres + pgvector)]
  end
  subgraph Worker["Worker (Python: Modal, AWS ECS Fargate, GitHub Actions or a Mac)"]
    direction TB
    V[validate] --> A[audio] --> T[transcribe] --> S[slides] --> ST[slide text] --> X[transcode] --> C[chunk] --> CH[chapters] --> E[embed]
  end
  PG -- "claim (SKIP LOCKED)" --> Worker
  Worker -- "video, transcript, slides" --> R2
  Worker -- "segments + sub-chunks" --> PG
  subgraph Ask
    Q[Question] --> EM[embed] --> M["match_segments (3 ranked lists, RRF): top 12"] --> G{relevant?}
    G -- no --> OL["try the course outline"]
    OL -- "it answers" --> ANS
    OL -- "it doesn't" --> NC[Not covered]
    G -- yes --> RR["rerank: 12 → 6"] --> LLM[Groq, Gemini fallback] --> VC[validate citations] --> RF[land on the sentence] --> ANS[Answer + chips]
    VC -- "invalid: one retry" --> LLM
  end
  PG --> M

Processing a lecture. When a lecture is queued the site wakes three cloud workers after it has answered the upload: Modal (16 cores, usually first), a Docker container on AWS ECS Fargate Spot, and GitHub Actions (the backup). Whichever starts first takes the job; the others find an empty queue and exit. The worker claims a queued lecture with FOR UPDATE SKIP LOCKED (Postgres is the job queue; there is no Redis) and runs one stage per file in worker/lecture_worker/stages/. Each stage leaves a marker, so a failed run resumes where it stopped. Speech is transcribed with Whisper large-v3-turbo (locally through MLX on Apple Silicon, or through Groq's hosted Whisper on the cloud worker). Slide frames are found by scene detection and read by Gemini. The transcript is cut into 60–90 second segments that end on a sentence, and each segment is also cut into ~20 second sub-chunks; both are embedded with gte-small (384 dimensions).

Answering a question.

1. The question is embedded by a Supabase Edge Function running the same gte-small weights the worker used (a parity test checks the two agree).
2. match_segments, one SQL function, ranks segments three ways (the segment's own vector, its best sub-chunk's vector, and full-text search) and merges the three lists with reciprocal rank fusion. It returns the top twelve.
3. If the best cosine similarity is under 0.78, the lectures' words are treated as not covering the question and no answer is written from them (step 8 is what happens next).
4. A small, fast model (Gemini's lite model) reads the twelve passages against the question and keeps the six most likely to hold the answer, best first. It has three seconds; if it is slow, over its free quota or replies with nonsense, the search order is used unchanged.
5. Groq (openai/gpt-oss-120b) writes the answer from those six segments only, ending every sentence with a citation. On a rate limit or outage the same prompt goes to Gemini.
6. Citations are checked in code, not trusted. Each must name a retrieved segment and a time inside it. An answer that fails is regenerated once with the reason; if it fails again the user gets an error, never an unverified citation.
7. Each citation is then moved from the start of its segment to the stretch of transcript that shares the most distinctive words with the sentence it supports, so the click lands on the sentence.
8. When the lectures don't cover the question (step 3, or the model says so in step 5), one more attempt is made from the course outline: its title, its lectures with their lengths, and their chapter names. That prompt answers only questions about the course itself, such as how many lectures it has or which one covers a topic, and its citations go through the same check. Anything else gets "Not covered in these lectures".

This is the beginning of the README. Read the rest at the original. Read the full README ↗