Nudge

A small agent that lives in your notch and does things on your screen.

Hold a key. Say what you want. It reads what you are looking at, finds the control, and clicks it — or takes the whole task away and finishes it.

Download · Build from source · How it works · Report a bug

---

The idea

Every AI assistant right now is the same shape: a text box. You describe what you are looking at to something that cannot see it, it describes back what you should click, and you go and click it yourself.

Nudge is on your side of the screen. It sees what you see, and it has hands.

You say	It does
"What is this error actually telling me?"	Reads the dialog in front of you and answers in plain language
"Where is the setting for scaling in this app?"	Finds the buried menu item and points at it, so you learn where it was
"Play something by Radiohead on YouTube."	Opens the browser, searches, picks a result, presses play
"Fill this form in with the details from that email."	Reads one window, types into another, field by field
"Teach me DaVinci Resolve."	Draws on your screen and walks you through it, one chapter at a time

No window. No dock icon. No chat history to scroll — just a cat in the dead space at the top of the display.

Install

Download the build — nudge.runmycrew.com serves signed, notarised .dmg for Apple Silicon and .deb / AppImage for Linux. It updates itself after that.

On first use Nudge asks for Screen Recording and Microphone. Grant both and relaunch — macOS does not hand a new grant to a process that is already running.

No API key? Sign in and turns go through our proxy. Prefer your own? Paste a Gemini or Anthropic key in Settings, or run a local model and pay nobody:

brew install ollama && ollama pull qwen3-vl:4b   # then set provider = "ollama"

Using it

Key	What happens
Hold ctrl alt	Talk. Release and it answers
Tap the same keys	Type instead, or skip to the next step
Draw while you talk	Circle anything on screen — it sees the drawing
esc	Stops it. Whatever it was in the middle of

A ring appears over the control to click and stays there until you actually click it, then the next step loads by itself. Turn on Click for me and it does the clicking too.

How it works

Each turn is a fresh capture, because step 2's target usually lives inside a menu step 1 opened — it is in no earlier screenshot. The model is asked what to do next and answers with exactly one of 31 outcomes:

Point · Launch · Open · Type · Press · Tour · Done · Unsure
Run · Write · Read · Edit · Fetch · Search · Task · Agent · Question · …

Each exists because collapsing it into another one caused a visible bug. Unsure is there because a model that may only point or finish will invent coordinates when shown a screen with no matching control. Question is what makes an unattended agent able to ask "which Sara?" instead of guessing.

Three things run underneath the loop:

• Grounding — on macOS the accessibility tree gives exact bounds for free; everywhere else, and inside Blender, DAWs and CAD where the whole window is one GPU canvas, the vision model is the universal path.
• A context budget — core/context.rs decides room for every section of the prompt first, then cuts each exactly once. A plain question used to ship 52,221 characters of prompt and now ships about 22,000.
• An agent runtime — core/run/agent.rs drives itself with a step budget, a visible stop, and a question it can ask you. While it runs it owns the real cursor, which is what "do it for me" means.

What it will not do

• Secrets stay in the Keychain. connections.toml only ever names one, as keychain:<item>. Nothing is pasted into a file or a commit.
• It refuses to look at password managers. core/screen/privacy.rs, mid-agent as well as before one starts.
• Counts, never content. The analytics send how long a turn took and which model answered. Never a goal, a transcript, a window title, an application name or a path — and the switch in Settings is off in one click. See core/counted.rs.
• Launching is validated as untrusted input. The app name comes from a language model, so paths, flags, shell metacharacters and URLs are all refused, with a test for each.

Layout

src-tauri/src/
  core/                 What Nudge does. No Tauri. Unit-testable.
    provider/           gemini · anthropic · ollama, behind one trait
    run/                session · agent · subagent — the loops
    screen/             capture · ax · click · keyboard · ink · privacy
    tools/              mcp · shell · files · fetch · relevant
    voice/              ear · record · transcribe · speech
    context.rs          the prompt budget          teaching.rs   tour scoring
    memory.rs           what it remembers          threads.rs    conversations
    counted.rs          what it reports            laps.rs       where time goes
  app/                  Wiring core to the OS. Thin by design.
    ui/ · input/ · commands/ · agent.rs · tour.rs · update.rs
  bin/                  errand · bench · picks · truth · probe · judge · ax
tests/layering.rs       Fails the build if core imports Tauri

ui/                     The panel and overlay. React, Tailwind v4, Vite.
server/                 FastAPI: releases, accounts, model proxy, counts.
web/                    The marketing site. Next.js.

The one architectural rule: core must not know Tauri exists. That is what lets 494 tests run without a window server, and it is checked rather than trusted — a documented boundary decays the first time somebody wants an AppHandle inside a provider.

Build it yourself

make tools                                           # one time
cp config.example.toml ~/.config/nudge/config.toml
make pin-identity ID="Apple Development: ..."        # macOS only — see below
make run                                             # build, sign, restart

make dev          # hot reload
make test         # 494 Rust tests
make lint         # clippy -D warnings, then fmt --check
make timings      # where turns have been spending their time
make bench        # the accuracy number

make on its own lists every target. CLAUDE.md has the conventions, the release steps, and the things that have bitten.

Code signing is not optional on macOS, even in development. TCC identifies an app by its signature, so an ad-hoc build gets a fresh identity every time and your Screen Recording grant silently stops applying — while the toggle still reads "on". .signing-identity pins the certificate. See CLAUDE.md for the release steps.

Linux is X11 only. Injecting input and reading a held modifier are things Wayland deliberately forbids; on Wayland the window appears and the hotkey never fires.

Proving it works

The whole product rested on one question: can a model return the on-screen position of a named control accurately enough to draw a ring around it? That is measured, not assumed:

make probe GOAL="open the UV editor"     # writes a PNG with the ring drawn on
make bench                               # scores every saved case
make timings                             # p50 per stage, per turn

Nearly every fix in this repository came out of one of those rather than out of reading the code. Spoken turns got roughly twice as fast when measurement showed the on-device transcriber was being abandoned after one cold-start timeout. Tour scoring promptly revealed that the sloppy boxes were mine, not the model's.

Status

Early, and honest about it. It works, it is genuinely useful, and it will also do things confidently and wrongly. 397 commits, 10 releases.

Not built yet, deliberately:

• Concurrent agents — two agents fighting over one cursor is incoherent
• Wayland — needs the RemoteDesktop portal and libei, which is its own project
• Windows — reachable through Tauri; the screen and input layers are the work

If you want to break it, that is the most useful thing anyone can do right now. There is a Report a bug button inside the app, or open an issue — what you asked it to do and what it did instead is the good stuff.

License

MIT. Do what you like with it.