Padawan

Teach Yoda what you know. Yoda teaches the next person, and stops them before they break a rule.

Padawan turns "watch me do it" into reusable skills. A Master shares their screen and talks through a task. A voice agent, Yoda, watches, stays quiet while the Master works, and asks why at natural pauses. The session becomes a Holocron: a saved skill with steps, the Master's reasons and guardrails. Later Yoda tutors a Padawan through that Holocron on their own screen and stops them before they break a guardrail.

Built for the Hack-Nation 7th Global AI Hackathon (2026), challenge 01 "The AI Apprentice" (powered by ElevenLabs).

Live demo: https://padawan-bay.vercel.app (sign up with an email on the login page; backend API docs at https://padawan.fastapicloud.dev/docs)

Glossary. Master = the expert who teaches. Padawan = the learner. Holocron = a saved skill. Jedi Archives = the library of published Holocrons. Yoda = the AI. Star Wars names and our own styling only: no official logos, stills, music or voice.

What it does

1. Teach. The Master shares a screen and narrates. The browser sends only the frames where the screen really changed.
2. Understand. A vision model describes what changed and returns structured events.
3. Ask why. At a natural pause (screen idle, Master silent, question budget left) Yoda asks about intent: why this tab, why option X over Y. He does not ask what the screen already shows.
4. Debrief. When the Master stops, Yoda asks the still-open questions and explains the process back (teach-back).
5. Holocron. The confirmed session is synthesized into a skill (steps, reasons, guardrails, quotes), reviewed, and published to the Jedi Archives.
6. Learn. A Padawan picks a Holocron and shares their screen. Yoda tutors by voice, asks for predictions, and shows a red STOP before a guardrail is broken. The lesson ends with a mastery report.

How it looks

Real captures of the live app at https://padawan-bay.vercel.app (header email painted over; static states, nothing recorded or published for the shots). Details in docs/screenshots/MANIFEST.md.

Dashboard: backend status, mic and Yoda preflight, Teach and Learn entry points.	Jedi Archives: published Holocrons with tags, guardrail counts and learner counts.
Holocron step: the decision, the Master's own "why", guardrails, and the question Yoda will ask the Padawan.	Teach: name the task, start recording.
Debrief: the steps detected from the session, checked before the Holocron is built.	Debrief: "What is still unclear" and the Yoda voice panel with the first open question.
Learn: the Padawan begins a lesson from a Holocron.	My learning: a lesson in progress, 2 of 4 steps.

Not captured live (they need screen-share and microphone prompts): an active recording, a Yoda voice session, the learn STOP banner and the mastery report. Those two screens exist as captures of the frontend's mock mode (NEXT_PUBLIC_API_MOCK=1, scripted data, not model output): STOP banner, mastery report.

Architecture

flowchart LR
    subgraph Browser["Browser: Next.js on Vercel"]
        CAP["Screen capture<br/>custom frame-diff gate"]
        VUI["Yoda voice UI"]
    end
    subgraph API["FastAPI on FastAPI Cloud"]
        FR["Frames endpoint<br/>one call in flight"]
        PLAN["Planner, segmenter, gaps,<br/>synthesizer, guardrails, reconcile"]
        VS["Voice sessions<br/>signed URL"]
    end
    VLM["Nebius token factory<br/>DeepSeek-V4.1-Flash VLM"]
    TXT["Nebius text model"]
    EL["ElevenLabs agents<br/>interviewer and tutor"]
    SB[("Supabase<br/>Auth, Postgres + RLS, Storage")]

    CAP -- "changed frames only (JPEG)" --> FR
    FR -- "frame + previous screen_summary" --> VLM
    VLM -- "events + new summary" --> FR
    FR --> PLAN
    PLAN --> TXT
    VUI -- "request signed URL" --> VS
    VS -- "signed URL" --> EL
    VUI <-- "live voice" --> EL
    Browser -- "login" --> SB
    API -- "user's JWT, RLS applies" --> SB

Notes on the arrows, checked against the code:

• The frame-diff runs in the browser (frontend/lib/frame-diff.ts), before anything is uploaded. Frames go straight from the browser to the backend, never through a Next.js route (Vercel's 4.5 MB body limit).
• The voice audio goes browser to ElevenLabs over a websocket. The backend only mints the signed URL and the per-session variables (POST /v1/voice/sessions); it does not carry audio.
• The backend calls Supabase as the user (their Bearer token is forwarded), so row-level security enforces ownership.
• A second, editable status map with what is built and what is not: docs/architecture.md.

This is the beginning of the README. Read the rest at the original. Read the full README ↗