House Twin Patrol

Qloo Agentic Hackathon (Duck Host), work in progress. This repo adds Qloo to House Twin, an existing agent: before the duck acts, it asks Qloo what each person in the room will like and shows the score. Live page: qloo.adtomic.co (web/). Everything before Sep 30 and the Nebius pass is the tag baseline-nebius; git diff baseline-nebius shows what changed for Qloo. The Qloo key stays on the server, never in this repo or the browser, and no Qloo response data is committed.

Walk through your home once with an iPhone. A small home robot then knows every room, takes spoken orders in any language, and asks you when it isn't sure.

· Live page: nebius.adtomic.co · Video: (YouTube link, Friday) · Nebius × NVIDIA Global AI Hackathon, Physical AI track · Jose Mauricio Carpio (solo)

"Revisa la cocina y luego el jardín." The robot hears it (Nemotron Omni), picks the two rooms of this home (Nemotron Ultra), says its plan aloud, walks there with its own sensors and reports back. The robot is Pollen's Microduck: its real software and trained walking policy, driving a simulated body inside a twin of a real flat.

The problem

A new home robot is lost: it has never seen your rooms. The usual fixes are stickers on the walls, beacons, or a long mapping run by the robot itself. And a robot maker can't test in its beta testers' real homes before the robot ships.

What it does

• One phone walk becomes a twin. The iPhone app records LiDAR depth, colour and pose while you tap each room; the server builds the floor plan, walls, doors, rooms, furniture and 3D, with each room's floor height.
• The robot finds itself in that twin from an 8×8 depth sensor: one turn on the spot, no stickers, no beacons.
• It takes orders in your words and language. NVIDIA Nemotron turns speech into the rooms of this home.
• It asks when it isn't sure: lost, an arrival in doubt, a vague or unsafe order, a step too tall for its body.
• Any robot body. The same twin loads in NVIDIA Isaac Sim with real floor steps; each body gets its own route.

Talk to it

Real replies from Nemotron 3 Ultra on the twin of the real flat (out/model_calls.jsonl, 29 Oct, NVIDIA API catalog):

You: "Revisa la cocina y luego el jardín." Robot (3.6 s): kitchen → garden. "Revisaré la cocina y luego el jardín."

You: "Is the fridge closed?" Robot (1.5 s): kitchen. "Checking the fridge in the kitchen."

You: "Check the balcony." Robot (1.6 s), asks: "This home doesn't have a balcony. Which room would you like me to check?"

You: "Go check the kitchen, I think I left the stove on." (NVIDIA Nova Carter, Isaac Sim) Robot, at the step: "I can climb 3 cm, but the floor steps 14 cm from passway into kitchen. Can you check kitchen for me?"

Who it's for

• Home-robot makers (Pollen's Microduck, LeRobot builds, any small robot with a cheap depth sensor): turn a beta tester's home into a test bench before the robot ships. One phone walk, no stickers on the walls, no mapping run.
• The end use: check-in rounds for someone living alone. "Is the stove off?", "Is someone in the garden?", asked in their own words and language.

What's real, what's simulated

real	simulated
The iPhone LiDAR walks of a real flat (RGB + depth + ARKit pose + room taps), the app, the upload, the twin build	The robot's body, its 8×8 depth sensor and camera, inside a MuJoCo twin of that flat
Pollen's robot software (robotd, tofd, mediad) and its trained walking policy	The person's answers in filmed walks are automatic unless said otherwise
Speech from a real microphone; every Nemotron call, logged with its latency	Furniture in the twin is boxes

Full NVIDIA stack, and where it runs

Nemotron 3 Ultra · Nemotron 3 Nano Omni · Nemotron 3.5 Super VL · Nemotron 3.5 Lightning · Isaac Sim 6.2 · OpenUSD · Nova Carter · Jetbot · RTX 4090.

Every model call goes through one client, agent/nebius.py (OpenAI-compatible), and is logged to out/model_calls.jsonl with provider, model id, latency and tokens. It is built for Nebius Token Factory. The numbers below were measured on NVIDIA's API catalog (same Nemotron models), because a Nebius account could not be opened from Peru during the hackathon. Switching is one setting: MODEL_PROVIDER=nebius scripts/nebius_rerun.sh re-runs every eval quoted here on Token Factory (see Switch to Token Factory).

model	job	when	code	measured (dev, NVIDIA catalog)
Nemotron 3 Ultra	spoken command → which rooms, in which order (tool calls over this home's house.json); the robot's questions; the patrol report	once per command, per question, per patrol	agent/language.py	commands: 91.4 % right vs 55.6 % for keywords; p50 1.9 s, $0.97 per 1000
Nemotron 3 Nano Omni	microphone audio → words	once per spoken command	agent/voice.py	p50 2.0 s, p95 4.9 s; 5 of 14 live calls failed (503)
Nemotron 3.5 Super VL	camera frame → which room	offline test on real frames	eval/vl_rooms.py	78.9 % vs 75.2 % for the colour vote: no real gain (see Limits)
Nemotron 3.5 Lightning	the same command test, cheaper	tested, not used	eval/commands.py	82.1 % right

This is the beginning of the README. Read the rest at the original. Read the full README ↗