BlinkFlow

Say it. Fix it. Prove it.

See a real run, no install: the purple-box incident, from the snapshot to the proof. It is the static report of examples/purple-box/.

Watch the 3-minute video: the same loop, said, fixed and proved, on the live simulator.

For the judges (5 minutes)

Criterion	What it shows	Where
Impact (30%)	Say what went wrong on screen, get a proven fix in the robot's code. Frozen-code pass (new seed, code frozen): 8/8 repaired and passed the incident and control checks; Wonder's cause named a planted line in 4 of 8, the kept patch touched one in 7 of 8. 7 of the 8 repeat first-pass mutations, 1 is new; pass of Oct 28, before the current gate and control rule. Live: contact in 5/5 runs before the fix, 0/5 after	Results, Where this could go
Use of Meta tooling (25%)	One job per model. Muse Glimmer (open weights, on this PC) watches and names. SAM 3.1 tracks. Muse Wonder finds the cause. Muse Code fixes. Muse Voice Transcribe listens. Every call is logged	How it works, models.json
Technical depth (20%)	Distances from SAM 3.1's masks. A diagnosis whose every citation is checked against the frames, log lines and code it was sent. An exact replay in a deterministic twin. Rollback and retry. A gate before each tool call	results/bug_suite.md, Safety
Demo (25%)	The purple-box incident from the sentence to the proof, as a page anyone can open, and the live console	The report, The console

Open weights: Muse Glimmer runs on this PC → provenance. Try it in ~2 min, no GPU → quickstart.

What is new here

• It watches pixels, not the simulator's insides. BlinkFlow looks at the window, as the developer does. Plain measurements come first: SAM 3.1's masks become distances and a first contact. Only then does Muse Wonder look, and its answer is checked against what it was sent.
• The proof is the same scene, played again. A deterministic twin replays it exactly, and it is measured again on pixels. A fix that fails is undone and tried again. Then the person answers "Keep the fix?".
• A referee the agent never sees, and a gate. Results are scored from contact logs that BlinkFlow never gets. The gate is a filter, not an isolation: it checks every Muse Code call against the target folder and refuses, before it runs, one that names a path outside it.

Beyond the duck: the same loop on a 3-joint arm

The loop is not tied to the biped. A hand-written MuJoCo arm (a 43-line scene, a 45-line controller) has a planted bug, clearance = 0.0, so it sweeps through a red block. The same SAM 3.1 measurement, Muse Wonder diagnosis, citation check, gated Muse Code fix and pixel proof run on it through thin adapters (blinkflow/arm/); as with the duck, Muse Code may touch only the robot's controller (blinkflow/arm/robot/), not the world or the harness, and the fix is accepted on a mixed basis: contact and the two control scenes are measured from pixels, and reaching the target (within 20 mm) is read from the simulator's own end-effector position.

Before: contact	After: target reached

Contacts 1 → 0, controls 2/2 clean, Muse Wonder's cause grounded with no bad references, and Muse Code's fix is one line, made on the first attempt (0.0 → 0.15). The finished loop cost $0.27. One limit: SAM 3.1 did not find the arm from text, so the arm is tracked from a box marked once on the first frame, the same before and after. The run is in examples/arm/ and the numbers and limits are in results/arm.md.

Two more runs, with the same loop and the same rule: a sign error in the controller's lift (fixed with one logic line on the first attempt) and the clearance bug driven by a real spoken sentence through Muse Voice Transcribe, same fix and proof. results/arm.md. All arm runs so far, from their ledgers: $1.86.

Contents

For the judges · What is new · The arm · The robot · How it works · Why the Meta stack · The code · Results · Try it in ~2 min · Run the full loop · Glimmer provenance · Limits · Pre-existing vs built · Safety

The robot in the demo

Microduck is a small open-source biped robot by Pollen Robotics / Hugging Face. Two pre-existing layers move it:

• The legs: Pollen's walking policy, a neural network trained with reinforcement learning in a MuJoCo simulation. It turns "walk at this speed, turn at this rate" into steps.
• The brain: microduck-brain, the author's own, picks those speeds from one 8×8 depth sensor facing forward. It walks ahead, steers away from what comes near, stops and turns in place at walls and boxes, turns out of corners, goes around low objects on the floor, keeps away from ledges, and remembers for a few seconds what it saw beside its body after it leaves the sensor's view.
• The five scenes test one rule each: the purple box (what passes beside the body), a low cube, a ledge, a corner, a narrow doorway.
• The demo bug is an earlier bug of the author's, put back: the brain judges "ahead" from the sensor's front columns only, and its side memory is off (side_memory_seconds = 0.0). The purple box leaves the sensor's view before the right leg has passed it, and the leg hits the box it should have steered around. In the example, Muse Code turns the memory back on (3 s).

How it works

For people who write robot software and test it in a simulator: you say what went wrong on screen, BlinkFlow finds the cause in the robot's code, fixes it, and proves the fix on the same scene.

BlinkFlow watches a robot simulator the way you do: through the screen. When the robot does something wrong, you say what you saw. BlinkFlow rewinds to that moment, works out which part of the robot's program caused it, writes the fix, and plays the same scene again to prove the problem is gone. If the fix doesn't work, it undoes it and tries again.

What it films is the simulator's window on your screen, not a camera on the robot. It runs Meta's open-weight Muse Glimmer on this PC: the rolling buffer and the continuous Glimmer checks stay local; when an incident is called, the incident clip goes to SAM 3.1, and selected frames, the log and the code go to Muse Wonder.

Step	What happens	Meta model
Remember	The last 60 seconds of the simulator window are kept locally	—
Watch	Or nobody says anything: about once a second the newest frame is checked locally for a robot touching, nearly touching, fallen or stuck; a raised hand is confirmed on pixels before the loop starts, and the moment comes from BlinkFlow's own clock (results/watcher.md)	Muse Glimmer (local), SAM 3.1
Listen	Hold F9 (or, with --tap-to-talk, tap the arm of the Ray-Ban Meta glasses, and tap again when done) and say what you saw ("three seconds ago its right leg hit the purple box"), or type it; the moment is the key press on BlinkFlow's own clock minus the time you said; the sentence is read locally (results/voice.md)	Muse Voice Transcribe, Muse Glimmer (local)
See	Objects in those seconds are named locally and then followed pixel by pixel; the masks become measurements (distance between objects per frame, first contact)	Muse Glimmer (local), SAM 3.1
Understand	Key frames + measurements + time-aligned logs + the robot's code → the cause, plus a plain-language explanation. Every claim cites its source: a key frame, a measurement, a log tick or a code line, and a check marks the diagnosis grounded or not: the file, function and lines exist, every key frame and tick it names was sent, and any reference that names none is listed in bad_refs. An ungrounded diagnosis is kept and shown as such; the fix is still judged on pixels, not rejected for it	Muse Wonder
Fix	A coding agent patches the code in the target folder, each of its calls checked by BlinkFlow's gate; a git commit is taken first	Muse Code (Python SDK)
Prove	The same scene, replayed exactly in a deterministic twin, and checked again on pixels; failed fixes are rolled back and retried. For the demo incident, then confirmed on the live simulator (results/live_check.md)	SAM 3.1, Muse Code

Why the Meta stack

• Muse Glimmer runs locally, with open weights. It watches all the time, 0 tokens billed, and the frames it watches stay on the PC; only when it raises a hand or you speak does an incident clip leave, to SAM 3.1.
• SAM 3.1 turns video into masks across frames, and the distance between the masks is the measurement: no model eyeballs it.
• Muse Wonder reasons over frames, log and code at once; Muse Code edits, each call checked by BlinkFlow's gate. The whole chain, from watching to the fix, is one ecosystem.

Capture and measurement work on any window; the rest does not. BlinkFlow watches pixels, through GPU screen capture (DXGI Desktop Duplication + NVENC), of the window named by the sidecar's --title; there is no plugin in the simulator. That capture and measurement part should apply to any simulator (Isaac Sim, Gazebo, CARLA, Unreal), but it has not been tested on them. Diagnosis needs the controller's source and logs, and the proof needs a deterministic replay adapter for the robot (two are written: Microduck and the arm).

Where it is in the code

• The 60 s buffer: sidecar/capture.py on Windows keeps the ring of 1 s segments; blinkflow/capture.py (Sidecar) starts it and cuts snapshots from WSL.
• Muse Glimmer: blinkflow/models.py (glimmer), called to name objects in see.py (name_objects), describe key frames in diagnose.py (build_frames), read the sentence in voice.py (parse) and watch in watch.py (Watcher).
• SAM 3.1 and the contact: blinkflow/sam.py (track) calls it, Meta's meta-sam-parser decodes the masks (masks.py keeps them, decoded, for the page), see.py (gap, base_of, measure) turns them into distances and a first contact.
• Muse Wonder and the citation check: (, ).

Results

Measured during the hackathon; each number links to the file it comes from.

What	Result	Source
Bug suite, stratified by layer (11 mutations drawn with a sealed seed, 8 gave an incident); blind to the mutation; perception rules were revised between passes	5/8 on the first pass, 8/8 on the third: incidents gone, control clean in all five scenes. Changes on the mutated line: 5/8 on the first pass, 6/8 on the third	results/bug_suite.md
Same suite, frozen-code pass: a new seed, the code frozen, one pass, no reruns. 7 of the 8 kept bugs repeat a first-pass mutation (same value), 1 is new; the pass ran before the current gate	8/8 repaired and passed the incident and control checks (incidents gone, control clean in all five scenes). Muse Wonder's cause named a planted line in 4/8; the kept patch touched one in 7/8. Pass of Oct 28, before the current gate and control rule	results/bug_suite.md
First suite (6 bugs), blind to the mutation, one pass	6/6 fixed; Muse Wonder's cause on the mutated line in 5/6	results/bug_suite.md
Live simulator, the demo incident, 20 s per run	contact in 5/5 runs without the fix, 0/5 with it (an earlier pass, with a CPU-heavy job beside it: 3/5, 0/5)	results/live_check.md
Seen → fixed and verified (machine time)	368.0 s for the example run (examples/purple-box/); 386.6 s for the run the film shows (Oct 31). Since Oct 31 "verified" includes the control: the clock stops only after the four other scenes are replayed with the change and come out clean. The 135.8–234.0 s over 10 purple-box loops (median 169.2 s, the spread mostly Muse Wonder, 55–120 s) come from earlier runs, before that change, when the clock stopped at the replay of the incident and the control ran after it

Every run writes a models.json: each model call with the model, where it ran (this PC, Meta, Baseten, the cache), tokens or frames, seconds and cost. Cost details (the SAM 3.1 cache, Muse Code's model, tools/spend.py): docs/RUNNING.md.

Try it in ~2 min, no GPU

A recorded live run of the demo incident is in examples/purple-box/: the 60 s snapshot, the sentence, the evidence, Muse Wonder's diagnosis, both answers, Muse Code's change, the replays before and after, the control scenes and the model calls. No GPU, no simulator, no key. A person tapped the arm of Ray-Ban Meta glasses, said "a couple of seconds ago the robot hit the purple box" into their microphone, and tapped again (said.wav); Muse Voice Transcribe wrote it down, and the same person answered yes to "Fix it?" (after 3.3 s) and to "Keep the fix?" (after 4.0 s), both in the console. That was run 20261031-235414 (386.6 s, $0.38), the one the film shows, kept at 9a585ea. Its control followed objects Glimmer named, and one scene's object was seen in no frame, which counted as clean; that rule changed on Nov 1, so the example was run again from the same inputs: the same clip (outside/), the same sound and key press (voice clip), with both questions answered automatically (--auto). Run 20261101-093835: seen → fixed and verified in attempt 1, 368.0 s of machine time, control included (4/4 other scenes clean, each against its own obstacle), $0.95.

1. The report. Python 3 (tested on 3.12), standard library only:

git clone https://github.com/samoxis/blinkflow && cd blinkflow
python3 -m venv .venv && . .venv/bin/activate
python3 -m blinkflow.console report examples/purple-box/events.jsonl --out report

Open report/index.html over a local server that answers byte-range requests, since the replays seek in their videos (python3 -m http.server does not, and they snap back to the start); the console's own server does, on the same example: python3 -m blinkflow.console serve --events examples/purple-box/events.jsonl --port 8000, then http://localhost:8000. Either way: the whole loop step by step, the masks, the distance to first contact, the cause with its key frame, log row and code line, the replay before | after, the clock and the bill.

2. Measure it again. The contact is re-derived from SAM 3.1's saved replies, offline (needs ffmpeg):

pip install -r requirements-judge.txt
cp -r examples/purple-box /tmp/pb
python3 -m blinkflow.see /tmp/pb/window/snapshot.json --out /tmp/pb/window/evidence --reuse

/tmp/pb/window/evidence/evidence.json comes out identical to examples/purple-box/window/evidence/evidence.json (first contact, closest gap, the gap in every frame) except for the two paths it names. It writes into the evidence folder, hence the copy.

What needs a key. Nothing above. Asking the models again does: SAM 3.1 and Muse Wonder need MODEL_API_KEY (Meta Model API), a diagnosis with Muse Spark needs BASETEN_API_KEY (Baseten), and both also need the robot's code. Keys are read from the environment or ~/.config/blinkflow/env, never from the repo.

Run the full loop (~30–60 min, GPU)

The loop itself on a new machine: BlinkFlow sees the incident, Muse Wonder finds the cause, Muse Code fixes the target, the twin replays the scene and SAM 3.1 checks it again. Step by step in docs/RUNNING.md:

1. Linux or WSL2, Python 3.12, ffmpeg, an NVIDIA GPU that holds Muse Glimmer (Ollama), the Muse Code CLI, MODEL_API_KEY.
2. pip install -r requirements.txt, then python3 -m blinkflow.gate install.
3. The target, python3 tools/make_target.py ~/duck-target, and Pollen Robotics' simulator with the demo scene (blinkflow-demo-scene).
4. One loop on the purple box, from the recorded clip (no sidecar) or live (Windows 11 + WSL2).

Checked on Oct 28 from a fresh clone, a new venv and a target made by tools/make_target.py (same files as the one every run used): the purple-box loop, without the SAM cache, fixed it in attempt 1, contacts 1 → 0, in 167.3 s for $0.83 (results/fresh_setup.md, which also says what was not redone from scratch).

Muse Glimmer, on this PC — provenance

Glimmer is the open-weight model that names objects, describes key frames and reads the person's sentence, on an RTX 4090 through Ollama. To check that without this PC, examples/purple-box/ carries:

The model: muse-glimmer, 27.9B, quantization Q4_K_M, projector clip 1.9B, served by Ollama 0.40.11, digest de878ce33ad8. The weights are Ollama's public library model muse-glimmer:30b-q4_K_M (also its latest), unchanged: the public manifest's SHA-256 is that digest, and its model and projector layers are the SHA-256 of the two weight files on this PC.

• glimmer-model.txt: ollama show muse-glimmer on this PC, the SHA-256 of both weight files, the public manifest, and the one curl | sha256sum that checks it without a GPU.
• models.json: every model call of the run. Glimmer's six are muse-glimmer, local, with tokens in and out and seconds (the sentence, the names, four key frames). Of the 52 SAM 3.1 calls, 30 were paid ($0.63 of the run's $0.95) and 22 are cached, $0, each with the hash of the video bytes, concept and parameters it answers.
• Ollama's own answer during the run (/api/ps, the glimmer.vram event in events.jsonl): 16.7 GB, all in VRAM, Q4_K_M, context 8192. Glimmer's six calls, with their times and tokens, are in events.jsonl and models.json.
• provenance.txt, from the Oct 31 run the film shows (voice live samples; a rerun from a clip does not): ollama ps (Windows) and nvidia-smi (WSL) every ~3 s from the start of the run to its end, all 145 samples muse-glimmer:latest, ID de878ce33ad8, 16 GB, 100% GPU, context 8192.
• glimmer.jsonl: Glimmer's raw replies from the run, each with what it was asked and the image it saw.

python3 tools/pack_example.py <run>/events.jsonl examples/<name> packs any run the same way: paths made relative to the example, the home folder written as ~, no keys, and never the scorer's ground truth.

The console

A local page that shows the loop live, one step at a time, with the model behind each step, from the last 60 seconds rewinding to the moment to the replay before | after, the clock and the bill. "Fix it?" and "Keep the fix?" can be answered there or in the terminal. python3 -m blinkflow.console serve (http://localhost:8777); the other commands (the static report, the film mode, the teaser, the side shot) are in docs/RUNNING.md.

Where this could go (not built or tested yet)

BlinkFlow needs three things: a screen, a log and a folder of code. Only the deterministic twin and the target are specific to the duck; the rest of the loop is not, so it could serve:

• Warehouse and delivery robots, tested every day in simulation before they touch a real shelf.
• Home robots, like a vacuum that keeps getting stuck under the same sofa.
• Open-source robots for hobbyists and schools (Microduck, LeRobot arms, Reachy), whose owners are not robotics experts.
• Game characters that walk through walls or get stuck in doorways: also pixels, logs and code.

On a real robot the proof changes: the real world never repeats exactly, but the robot's recorded sensor log can be replayed through the fixed brain ("with exactly what it saw, would it turn now?"), then checked in simulation, then on the robot, and only with a person's "Keep the fix?".

When there is no simulator to replay in, the same recorded sensor inputs (here recorded in the simulator, not on a robot) can be replayed through the brain before and after a fix (blinkflow/sil/, built Nov 1). On 4 of the 8 suite bugs the fixed brain brakes 2.85–8.2 s before the recorded impact while the buggy one keeps walking; on the other 4 and on the purple box the result is inconclusive (the first difference is a small steering change). Open-loop decision evidence only, not an outcome proof: results/sil.md.

Limits

• The twin is not the live simulator. It starts the robot standing (no sit-to-stand boot) and re-states robotd's controller in Python, so robotd itself is bypassed.
• The live simulator is not deterministic. Four wall clocks run it, so the same brain does not walk the same path twice. That is why the proof is double: exact in the twin, then 5 runs on each side live.
• Muse Wonder varies. The same evidence gives different lengths of answer (4.7k–9.1k output tokens), and that alone moves the loop time by up to ~50 s; its cause is not always on the mutated line (results/bug_suite.md).
• The watcher confirms after the fact, a median 19.5 s after the incident (13–52 s): it is a safety net, not a brake.
• "Stuck" has no real case. The watcher checks for it, but no run has had a robot stuck.
• One real spoken run so far: the example's sentence was said through Ray-Ban Meta glasses (Bluetooth, tap-to-talk) and transcribed correctly, one word doubled. The 5-sentence accuracy table is still from TTS; how far people's "N seconds ago" lands has one data point.

Built during the Meta Global AI Developer Hackathon, Oct 26 – Nov 4, 2026. Indie Developer track, theme: Push the Frontier. The build log is in DEVLOG.md.

New vs pre-existing: BlinkFlow's implementation was built after the kickoff on Oct 26, 2026; evidence bundles include declared pre-existing target snapshots (NOTICE files); the robot, its simulator and its brain existed before — see Pre-existing vs built during the hackathon.

Pre-existing vs built during the hackathon

The rules ask entrants to separate earlier work from new work. BlinkFlow's implementation was built after the kickoff session on Oct 26, 2026. Evidence bundles include declared pre-existing target snapshots (NOTICE files in examples/purple-box-sil/ and examples/suite-sil/).

Existed before, used only as the test target, not part of this repo:

• microduck-brain, an autonomy layer for the open-source Microduck biped (Apache-2.0), by the same author. BlinkFlow works on a clean copy of it (no README, no tests, no history, hint comments rewritten), so it cannot read the answer.
• The Microduck MuJoCo simulation and walking policies by Pollen Robotics / Hugging Face (Apache-2.0), and a recoloured demo scene.
• The recoloured demo scene with the purple box, the hook that shows the simulator in the browser (Viser) and the camera-direction script that films it (all Oct 22). Published unchanged, after the kickoff and labelled as pre-existing, on the branch blinkflow-demo-scene of the author's microduck_rl fork: scene_demo.xml, apartment_demo.xml, duck_viser_hook.py + .pth, duck_cam_director.py and the incident's camera script shots_incident.json.
• The demo bug in the brain: the author's commit "demo: reintroduce finding 6" (Oct 22), never pushed. tools/make_target.py re-states it as text edits.
• The script that first made the target (strip the README, tests and hint comments). Not in this repo. It was started before the kickoff (Oct 26, 14:42) and its list of hint comments was last edited at 19:32 that day. tools/make_target.py replaces it: rewritten from scratch on Oct 28, no code copied, and it makes the same target file for file.
• Preparation tests made before the kickoff on Oct 26: a first API call, SAM 3.1 tracking on an old clip, a headless Muse Code session, a Muse Glimmer test and a prototype screen buffer. None of that code is included; the buffer was rewritten from scratch here.

The arm uses the same MuJoCo (3.10.0) as the duck's simulation, declared above. Its scene (blinkflow/arm/world/), controller (blinkflow/arm/robot/), twin and adapters (blinkflow/arm/) and its example (examples/arm/) are new work from Nov 1.

The log replay uses microduck_brain.record.replay, part of Pollen's pre-existing brain, unchanged. The adapter, measurement, classification and tests (blinkflow/sil/) and its examples (examples/purple-box-sil/, examples/suite-sil/) are new work from Nov 1.

Honesty notes

• The simulator's own contact log is never given to BlinkFlow. It is used only as ground truth to score whether a fix worked.
• The twin loads the scene to simulate it; perception, diagnosis and the fix receive only pixels, the brain's own log and the code — never the contact log, truth.json or the scene geometry. The arm's run.py reads truth.json only as an eligibility guard on the incident (a real contact, and pixels and physics agreeing within 0.1 s) before diagnosis, not as input to the models.
• Results are measured during the hackathon, on BlinkFlow's own fixes, including bugs injected blind into the robot's code.
• Every incident report lists each model call: model, where it ran, tokens.
• Speed-ups in the demo video are labelled on screen.
• Grounded in the purple-box and arm clearance runs; not grounded in arm-bug2 and arm-voice (arm-bug2: categories cited instead of frames/ticks; arm-voice: a real tick but no key frame and two references to nothing), both still fixed and proven on pixels.
• The arm (blinkflow/arm/, examples/arm/) was built on Nov 1 with Codex (GPT-6.1 Sol), in a separate repo from a written brief, and moved here. So was the log replay (blinkflow/sil/, examples/*-sil/).

Safety

Muse Code runs under muse serve's default profile (not :unrestricted), which does not stop a read or a write outside the target, so a PreToolUse hook (muse-gate/, python3 -m blinkflow.gate install) refuses any tool call that names a path outside the target before it runs. That hook is a filter that checks every Muse Code call against the target folder, not an isolation: it reads the paths a call names, so code that reaches outside without naming the path (a script that builds one as it runs, say) is not stopped by it. Every fix starts from a git commit; BlinkFlow asks "Fix it?" before Muse Code writes and "Keep the fix?" after the proof, and a no rolls the change back. How it was measured, and how to undo a kept fix: docs/SAFETY.md.

Credits and licence

The icon was made with Meta's Muse Image; provenance in results/icon/.

The deterministic twin (blinkflow/twin.py) is new code, but the physics, the scenes, the depth sensor and the walking policies it runs are imported from Pollen Robotics' Microduck (microduck_rl), not copied.

This repository is Apache-2.0 (see LICENSE). SAM 3.1's masks are decoded with Meta's meta-sam-parser 0.1.0 (SAM License), installed as a dependency (requirements-judge.txt), not included here. Microduck and its simulation are by Pollen Robotics and Hugging Face, Apache-2.0. Code in this repository is written with AI coding assistants under the author's direction.