Clio
Every lesson trains two students: you, and a robot.
Clio is an AR tutor for Meta Quest 3, built with WebXR, that can teach you anything. Say what you
want to learn ("refill the windshield washer fluid"). An LLM plans the lesson from your goal and
what the headset camera sees, including a description of what to point out in each step. An
object finder (Gemini, or jev) finds those things in the camera view and Clio outlines them, then
shows each step with solid demonstration hands and animated motion cues while a voice coach walks you through it and catches
mistakes. Every
session is recorded as intent-labeled hand, object and language data that robots can learn from,
and it exports straight to LeRobot.
pnpm install
cp .env.example .env # add GEMINI_API_KEY
pnpm dev # server :8787 + headset app :5173
packages/protocol shared contracts (zod → TS types + JSON Schema)
apps/headset the WebXR app (three.js); domain logic is framework-free and unit-tested
apps/server Gemini relay, planner, vision, episode ingest (Node + Hono)
pipeline validate, Rerun replay, LeRobot export (Python + uv)
tracker real-time object tracking for the finder: YOLOE-26 + ByteTrack + jev (Python + uv)
procedures curated golden lessons (shirt fold, pH indicator)
Visualizer / desktop dataset inspection
The imported browser visualizer lives in visualizer/. Read
visualizer/GOALS.md for the product intent and agent handoff, and
visualizer/INTEGRATION.md for how it should adapt to Clio's existing
lesson recorder, local episode API and Python LeRobot exporter. The draft recording format is
included as design context; @clio/protocol remains the source of truth.
pnpm visualizer:install
pnpm visualizer:dev # loopback :3001, currently the public G1 LeRobot demo
pnpm visualizer:check # independent Bun/Next.js checks
Open /clio for local recordings and replay. Select finished episodes, choose a dataset name and
frame rate, then export and download a LeRobot v3 ZIP (human hand targets, tasks and object positions;
no video yet). The local server needs uv; export installs the pipeline's LeRobot extra as needed.
Private episode inspection and exports stay local, with no automatic publishing.
The visualizer has its own Bun dependencies and checks,
separate from the pnpm workspace.
