Second Hands

---

Mount any phone above your work: a phone repair, a cutting board, a flat-pack bookshelf. Second Hands finds the real guide on the web, turns it into steps a camera can check, watches every step, and speaks up only when it matters:

🔍 Finds the guide	👁 Watches your hands	🗣 Speaks up when it matters
Tavily pulls real guides at runtime. Nemotron Super VL turns them into steps a camera can verify.	Nemotron Super VL checks each frame against the step you're on and boxes what it's looking at.	Lightning decides whether to interrupt. Ultra double-checks the evidence before it ever says "Stop."

📸 See it work

🧩 The problem

One wrong move in a hands-on job costs real money, or a finger, and nobody is watching:

• 🔋 Unplugging a phone's display while the battery is connected can short the board.
• 🔩 A long screw in a short hole can crack the board underneath.
• 🔪 Flat fingers in front of a knife is how most kitchen cuts happen.
• 🪚 A bookshelf without its anti-tip strap can fall on a child.

Manuals and videos can't see what you are doing. An expert next to you can, but experts are expensive, and a repair shop owner can't watch every apprentice all day. Second Hands is that expert, sitting above the bench.

Built for phone repair apprentices first (shaped with input from a working phone repair shop), where one slip costs a customer's phone. The same loop works for any hands-on task with a guide:

🔌 Electronics	🍳 Cooking	🪚 Furniture	🏠 Home fixes	🚲 Bikes	✂️ Crafts
battery first, no metal on connectors	claw grip, cut away from your hand	panels the right way round, anti-tip strap	tap cartridges, drywall patches	flats, brakes, chains	soldering, sewing

Tasks where a missed warning could seriously hurt someone (mains electrical, gas, vehicle brakes, firearms) are refused.

✨ What's in the box

📱 The phone app (installable, no app store)

• Full-screen HUD over the live camera: segmented step bar, scan line, and a focus box drawn from the vision model's output
• Pair in a second: scan the laptop's QR (native BarcodeDetector, jsQR fallback) or type the 8-character code
• Camera check before starting (light, focus) and a flashlight toggle (Android)
• Hands-free voice commands: "skip", "back", "repeat", "check", "I'm right", "pause". It never reacts to its own voice.
• Manual controls: Back · Check now · Skip · Pause, so the person stays in charge
• Spoken warnings with a chime before every Stop; the screen stays awake

💻 The bench dashboard

• Procedure checklist, live view about 1 s behind the camera (a small preview stream that never touches a model), evidence panel (the frame next to the manual step), console-style event log
• Agent trace: every Token Factory call with stage, model, latency, tokens and cost
• Session report: steps done, mistakes caught, false alarms, duration, cost, and a timeline with the frame behind every warning

🧪 Built to be judged

• /try: one click, no camera needed. It plays a clearly labelled recorded run while the real-footage sample clip is being recorded, then runs the live pipeline on that clip (with the replay as a fallback if the daily demo budget is used up)
• Per-IP limits, a daily budget cap and a 5-minute cap per live session keep the public demo up through judging

⚙️ How it works

	Step	What happens
1	Start · /start	Pick a category and type the job ("Redmi Note 8 battery replacement", "Dice an onion safely"). Tavily finds real guides; Nemotron builds the checklist with what "done" looks like from above, the costly mistakes to watch for, and where to put the camera.
2	Pair · /cam	Scan the QR with any phone (or install the app). Camera check, then it starts watching.
3	Watch	The phone sends a frame whenever the scene changes. The dashboard shows the live view, checklist, events and every model call.
4	Catch	The same mistake in 2 of the last 3 frames → Lightning decides if it's worth interrupting → on high-severity steps Ultra verifies → the phone chimes: "Stop."
5	Ask	When unsure, it asks instead: "Is that the long screw? Hold it closer." I'm right silences it for that step.
6	Report · /report	Steps done, mistakes caught, false alarms, duration, cost, and the evidence timeline.

The catch loop, from frame to spoken warning:

flowchart LR
    F["📷 Frame<br/>(on change)"] --> VL["👁 Super VL<br/>observe"]
    VL --> C{"2 of last 3<br/>frames agree?"}
    C -- no --> F
    C -- yes --> N["⚡ Lightning<br/>warn · ask · silent"]
    N -- silent --> F
    N -- "high severity" --> U["⚖️ Ultra<br/>verify"]
    N -- "low / medium" --> A["🔔 Ask or warn"]
    U -- confirmed --> S["🛑 STOP"]
    U -- "not sure" --> A

📱 Works on any device

Everything runs in the browser over the internet, so the camera and the dashboard don't need to share a Wi-Fi network, and nothing is installed from an app store.

Camera	Dashboard
Android phone (Chrome)	Laptop / PC, or another phone	✅ Tested on a Poco F1 and a Moto G85
iPhone (Safari)	Laptop / PC, or another phone	✅ Tested in WebKit with iPhone 15 emulation: QR pairing, frames, live view
Laptop webcam	The same laptop	✅ Point the webcam at your hands
No camera	Any browser	✅ /try sample run

On iPhone, add it with Share → Add to Home Screen. Safari doesn't let web apps control the flashlight, and voice commands depend on the iOS version; the on-screen buttons always work.

This is the beginning of the README. Read the rest at the original. Read the full README ↗