pip install AutoCua
https://github.com/user-attachments/assets/f02ac3e9-fe6c-42f8-a685-88001d8d178e
See everything AutoCua can do → agent_capabilities.md
| Windows | Linux | macOS |
|---|
| Computer use — any desktop app, with real mouse and keyboard input | ✓ | ✓ | ✓ |
| Browser skills — Chrome, Firefox, Edge, Brave and Opera | ✓ | ✓ | ✓ |
| Coder — a coding agent in a real terminal | ✓ | ✓ | ✓ |
| Browser — your own Chrome, driven through the AutoCua extension | ✓ | ✓ | ✓ |
| Mobile use — an iOS Simulator, or your iPhone or iPad, over USB, or Wi-Fi in the app | — | — | ✓ |
| Mobile use — an Android emulator, or your Android phone, over USB, or Wi-Fi in the app | ✓ | ✓ | ✓ |
Browser agent: fast mode
6.1 seconds for a flight search. Agent mode: browser.
https://github.com/user-attachments/assets/8cc797d5-5556-40a9-bbfc-5d6aa8880fb8
Browser agent: quality mode
Under a minute for three tasks in one run: an iPhone 18 Pro into the Apple bag, a case into the
Amazon basket, and a Google Sheet of Meta's five-year financials. Agent mode: browser.
https://github.com/user-attachments/assets/82bf6163-55c9-4cc4-95e7-9c6466f2a25d
iOS agent: iPhone use
A paired iPhone, driven over the USB cable, or Wi-Fi in the app. Agent mode: mobile use, iOS.
https://github.com/user-attachments/assets/3c22eb45-d939-4742-9053-22ecc75b416f
Pick a surface
One string decides what the agent drives. Everything else stays the same.
from AutoCua.agent_launcher import run_agent
run_agent(mode="browser", provider="openrouter", model="gemini-3.8-flash",
task="Open wikipedia.org and summarise today's featured article.")
mode | What it drives | Where it runs |
|---|
"computer use" | The desktop. Clicks, typing, hotkeys, apps, AppleScript | macOS, Windows or Linux, auto detected |
"computer use, daisy chain" | The same desktop, plus the phone sub-agents when a phone is connected: android_agent, and ios_agent on a Mac. Plain "computer use" lists them offline | macOS, Windows or Linux, auto detected |
"coder" | A coding agent, straight into a real terminal | macOS, Windows or Linux |
"browser" | Your own Chrome through the AutoCua extension; headless runs on a debugging port | Any host with Google Chrome installed |
"mobile use, ios" | An iOS Simulator by default, or your paired iPhone or iPad over the cable | macOS host |
"mobile use, android" | An Android emulator by default, or your Android phone over the cable | macOS, Windows or Linux host |
From a script a phone always goes over the cable. Wi-Fi is the desktop app's: see
iPhone and iPad and Android. A phone needs device="hardware";
the default, device="simulation", is the simulator or the emulator.
On Linux, computer use needs a GNOME desktop (tested on Wayland), and computer use and the coder
both need a few distro packages that pip cannot install. See Linux, from pip.
Why it holds up on real software
No pixel guessing. The model never sees a coordinate. It picks a numbered element from a
view that fuses the operating system's accessibility tree with OCR, and a number it was never
shown is rejected before anything moves.
No debugging port on your desktop. Input goes in at the operating system's own input
layer. Mouse events posted at kCGHIDEventTap on macOS, pywinauto's real input path on
Windows, and a kernel driver when Windows UAC blocks everything else. An ordinary
application sees the same events it would get from your hand on the mouse.
No debugging port on your browser either. A normal "browser" run drives your own Chrome
through the AutoCua extension, which Chrome loads at launch with no store install. Chrome shows
its own "AutoCua started debugging this browser" bar while the agent works, and its Cancel
button hands the browser back to you. A debugging port is opened only for headless runs, on a
browser of the agent's own. The page scanner never calls Runtime.enable, and the
numbered boxes are painted into the screenshot, not into your DOM.
No context wall. A separate compression agent watches the transcript and splices a handoff
summary into it mid-run, so a long task finishes instead of dying at the context limit. In the
app you watch the memory ring's count fall.
No telemetry. AutoCua sends nothing anywhere except to the model provider you chose, and
to Telegram if you connect it yourself.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
