Computer Use

By Codewhale · macOS beta (notarized app and source) · Windows experimental unsigned preview · Linux experimental: X11 verified on a virtual desktop (Xvfb + openbox) and in the Docker test image, from source; Wayland input on sway (verified on headless sway), other Wayland desktops observation only

The 0.12.0 release provides the notarized universal Mac app and an unsigned Windows x64 preview. The Windows ZIP bundles Node and includes setup instructions; it is intended for testing, with physical-device, mixed-DPI and signing acceptance still open.

Let Codewhale see and operate your apps — through whichever interface fits each step: app scripting (AppleScript/JXA) where apps ship a dictionary, accessible controls, text entry, click, scroll and capture through the same MCP tools. The macOS helper keeps permissions and human controls in one menu-bar app.

• Set up once. See Accessibility and Screen Recording status, open the right Settings pane, then run a check in a disposable practice window.
• Keep working. macOS selects apps in background mode by default. Accessibility actions and process-directed typing remain available. Actions that need a keyboard-focus lease refuse before delivery; use browser control, a separate computer, or explicit foreground selection of the selected app for those tasks. Existing authorization covers foreground selection unless the person excluded it; human-input yielding and Pause/Stop remain.
• Stay in control. See selected apps and their input modes. The 0.13.0 candidate adds a compact Codewhale controller while an app is in use. Pause cancels queued work and releases held input; Stop ends existing sessions. Only the person using the menu-bar controls can allow input again.
• Update deliberately. The app checks for stable releases on request. Updates verify the digest, Codewhale signature and Apple notarization before replacing the app, and retain the previous install for rollback.

Setup page · Setup and troubleshooting · Release notes · Distribution · Background demo · Contributing · Security

Install

Install the helper and connect your host, or build from source:

1. Mac app (easiest). Download the disk image from the latest stable GitHub release, drag it to Applications, and open it. Open Computer Use… from the whale menu-bar icon, grant what it asks (Accessibility, then Screen Recording), and run the background check. No terminal, no Node, no compiler needed.
2. Inside Codewhale. Install, review and enable Computer Use as an ordinary plugin through Connectors, using a stable published plugin release. Plugin updates are independent of Engine updates. On macOS, install the separate Computer Use app above; it owns permissions and the human controls. The portable computer-use.tar.gz asset for this installation path is planned for the 0.13.0 release and is not yet published. Run request_access and follow its setup hint; a missing helper returns app_setup_required.
3. From source. Clone, test, install:

   git clone https://github.com/codewhale-hq/codewhale-cu-plugin
   cd codewhale-cu-plugin
   npm test                # unit + protocol, no GUI input performed
   npm run build:app       # dist/{macos,linux,windows}
   npm run install:app     # puts the app in place, registers it, opens it once
   

   Needs Node 20+ and, on a Mac, Xcode Command Line Tools. Then wire your host from the table in Developer quick start.

Verify. Ask your host for request_access. It identifies the computer, helper/version, available tools and missing permissions without sending input. via: "app" means grants belong to the Computer Use app; via: "direct" means they belong to the hosting app or terminal. Follow that receipt's appHint. macOS requires the separately installed helper; direct mode is an explicit developer override. A missing or disconnected helper is never permission to bypass its controls. Permission recovery is in Setup and troubleshooting.

Trust Codewhale once (0.13.0 source candidate). The first request to read or operate this Mac opens the helper's native approval panel. Choose Trust until revoked… and confirm with Touch ID or your Mac password, or choose Deny. Codewhale proves its installation pairing on later connections, so changing apps, tasks or models does not ask again. Hosts without a persistent pairing key receive approval for that connection only. OS permissions and specific consequential actions remain separate decisions.

This needs both an updated MCP plugin and a helper advertising host trust. Update the plugin through Connectors and the native app through its updater, then restart their connections. Older clients can retain their own consent cards or per-app gates. A signed candidate passed human approval, reconnect after helper restart, background input/capture and human revocation on one maintainer Mac. The revised controller and release packaging still need qualification. These candidate features are not claims about the published 0.12.0 app.

Choose your surface. Apps on this Mac use Computer Use; your signed-in Chrome uses Codewhale for Chrome; a clean browser uses Computer Use's browser tool. /computer status checks the first route; Chrome has its own connector and per-site grants.

Operating guidance. The server sends its concise packaged skill in standard MCP initialize.instructions. Hosts that consume that field can supply it before the first action. resources/list / resources/read expose the exact same guide plus detailed operations, refusal recovery and recording references. This supplies guidance; host-side delivery/logging and approvals remain the host's responsibility. Engine support is maintained separately.

Capture identity. Current source returns raster_id on screenshot, zoom and OCR receipts. Repeat it in pixel targets and parent zooms; newer captures refuse older pins before input. Nested zooms crop the bound parent file. The packaged guide teaches this through the existing MCP instructions/resources. This does not establish that the UI stayed unchanged or that the current source shipped in the installed helper. Cua comparison and remaining parity records the upstream reference and proof gaps.

Status. Release status is recorded in CHANGELOG.md. The notarized Mac app is published on this repository's GitHub releases (stable release: v0.12.0; this checkout prepares 0.13.0); remaining gates are tracked in the release checklist. The setup page offers the notarized disk image directly.

The native setup panel, background check and updater require macOS 13.5+ for the self-contained bundle. The source MCP server includes experimental Windows and Linux backends, with HarmonyOS devices over hdc. Windows ships only as an unsigned x64 preview ZIP and Linux remains source and Docker. The Linux source candidate adds an approval and Pause/Stop window; neither platform has qualified exact-window targeting or a signed installer, and both are excluded from the plugin's public host eligibility until those gates pass. Their raw input uses the shared desktop and must not be treated as background control. Windows semantic mutations bind the observed window and element runtime identities and refuse stale paths. See the publication review and porting plan. The SSH route remains experimental. See the platform-specific limitations.

Linux approval setup (source candidate). The Linux helper uses a GTK 3 control window and a system authentication prompt for each app approval. It requires Python 3 with GTK 3 bindings, polkit, and a desktop authentication agent. After building and installing the app, an administrator can install its bundled policy using the default installation path:

sudo install -o root -g root -m 0644 "$HOME/.local/share/codewhale-computer-use/plugin/app/linux/net.codewhale.computer-use.policy" /usr/share/polkit-1/actions/net.codewhale.computer-use.policy

If XDG_DATA_HOME was changed, use the installed app's policy path instead. Quit and reopen Computer Use after installing the policy. If the policy, GTK, or an authentication agent is unavailable, approvals fail closed. The session must be local and active. After authentication, the controller minimizes before granting access, and remains available from the desktop's window controls. If the desktop cannot confirm it minimized, the request is refused. The helper does not offer macOS-style remembered host trust on Linux. This controller has only been exercised on a disposable Linux desktop; real distributions and desktop environments still need qualification.

The server works with MCP hosts including Codewhale, Kimi Code, Claude Code, Codex CLI and Cursor. It has zero runtime npm dependencies. Developer checkouts use Node 20+; the macOS distribution includes a pinned Node 24 LTS runtime. Codewhale still reviews, trusts and enables a plugin through its existing Engine authority before the model can use it.

The Mac app

The notarized universal app ships as a drag-to-Applications disk image on this repository's GitHub releases, only from a published stable release whose release.json receipt qualifies; the ZIP archive on the same release is what the in-app updater installs. The plugin marketplace links there, and the setup page offers the image directly. Release status is recorded in CHANGELOG.md. To build and install from source instead, use door 3 in Install. See docs/DISTRIBUTION.md for the packaging, notarization and release procedure.

Developer quick start

Choose the appropriate setup route (Install), then point your host at the server:

Host	Configuration
Codewhale	Discovers the Agent Plugins v1 bundle (plugin.json + mcp.json); nothing to configure.
Kimi Code	Run /plugins install /path/to/codewhale-cu-plugin, then /reload. The native kimi.plugin.json registers the server and skills.
DeepSeek Harness (dsh)	dsh plugin --profile web add @codewhale/computer-use-dsh, then link skills/ into ~/.dsh/skills. See DeepSeek Harness.
Claude Code	claude mcp add computer -- node /path/to/mcp/server.mjs
Codex CLI	~/.codex/config.toml: [mcp_servers.computer] command = "node" args = ["/path/to/mcp/server.mjs"]
Cursor / Windsurf / VS Code	{"mcpServers":{"computer":{"command":"node","args":["/path/to/mcp/server.mjs"]}}}
Gemini CLI	same JSON in ~/.gemini/settings.json
opencode	{"mcp":{"computer":{"type":"local","command":["node","/path/to/mcp/server.mjs"]}}}

/path/to/mcp/server.mjs can be this checkout or the stable copy inside the installed app (install:app prints it — on macOS ~/Applications/Codewhale Computer Use.app/Contents/Resources/plugin/mcp/server.mjs). npm link also gives you a codewhale-cu command for hosts that want one.

Kimi Code

Install the desktop helper above first, then use Kimi's /plugins install command with this checkout's absolute path. Review the local plugin and choose Trust and install. Kimi copies it into its managed plugin directory and enables its MCP server. Run /reload in an existing session, then /mcp: plugin-codewhale-computer-use:computer should show connected and the tool list (37 advertised in 0.9.0; counts vary by host). /plugins info codewhale-computer-use shows the installed version and status. This flow was verified with Kimi Code 0.41.0. See the Kimi plugin documentation for managed updates and removal.

The server needs Node 20 or newer on Kimi's PATH and uses the installed Codewhale permission helper. Existing MCP servers and model settings are preserved. Models without vision can read the default text observations and request local macOS OCR; models with vision can also request screenshots.

DeepSeek Harness (dsh)

Install the desktop helper above first, then add the bundle in integrations/dsh. dsh manages a profile's plugins as npm packages: dsh plugin forwards to pnpm in the profile directory and then reconciles dsh.profile.bundles, so one command installs the package and adds its patch layer to the profile.

dsh plugin --profile web add @codewhale/computer-use-dsh

Use the path to this checkout's integrations/dsh instead of the package name to install from source. The tools arrive as mcp__computer-use__<name>; the bundle runs the server from the installed app with the Node already running dsh, so nothing else needs configuring. Set CODEWHALE_CU_SERVER to point at a /Applications install or a source checkout's mcp/server.mjs.

The two skills install separately. A profile whose agent presets own skill discovery — the web profile is one — disables the host-level skill-filesystem row, so a bundle cannot reach it. The user skill root is read in every composition:

PLUGIN="$HOME/Applications/Codewhale Computer Use.app/Contents/Resources/plugin"
mkdir -p ~/.dsh/skills
ln -sfn "$PLUGIN/skills/computer-use" ~/.dsh/skills/computer-use
ln -sfn "$PLUGIN/skills/recording"    ~/.dsh/skills/recording

To scope desktop control to sessions that ask for it rather than every session on the profile, put the same mcp-computer-use row in an agent preset instead of installing the bundle: duplicate standard in the dsh UI, then add the row from integrations/dsh/cordis.patch.yml to the copy's agent.cordis.yml and set customSkillDirs on its skill-filesystem. A preset whose rows cannot be resolved is listed as broken with the reason, so a typo shows up in the picker rather than at session start.

Verified with dsh 0.1.5-rc.1 and plugin 0.6.0 on macOS: dsh plugin add into a profile, 44 tools bridged, both skills in the catalog, and a live computer_list call returning from the helper. Remove it with dsh plugin --profile web remove @codewhale/computer-use-dsh. Models without vision read the default text observations and can request local macOS OCR; models with vision can also request screenshots, which dsh bridges as attachments.

The desktop app

macOS grants Accessibility and Screen Recording to the responsible process of a permission check. A bare node mcp/server.mjs inherits its host's identity, so the grant lands on Terminal, Cursor, or Claude, and every host needs its own. The app fixes that:

• What it is — a tiny native launcher (app/macos/launcher.c on macOS) that runs app/daemon.mjs, a long-lived process that executes the platform backend and answers one-line JSON requests over a per-user socket (~/.codewhale-cu/app.sock, a named pipe on Windows). Only the same allow-listed tool set the ssh agent accepts will execute; the socket is never a shell.
• How the server uses it — every call on the local computer goes through the app when it is running. If it is installed but not running, the server launches it (via LaunchServices on macOS, so it is its own responsible process) and waits for it. On macOS, a missing helper returns app_setup_required with instructions to install and open the separate app; a registered helper that cannot start fails closed. request_access reports which mode is active (via: "app" or "direct"). Linux and Windows can run directly in the server process when no helper is installed. CODEWHALE_CU_APP=off is an explicit developer override for direct mode, never an agent workaround for the person's Pause or Stop choice.
• First launch — open the whale menu to review permission status. Only the setup buttons request grants; starting the helper does not prompt automatically.

Where install:app puts things:

OS	App	Also
macOS	~/Applications/Codewhale Computer Use.app (registered with LaunchServices)	--login: ~/Library/LaunchAgents/net.codewhale.computer-use.plist
Linux	~/.local/share/codewhale-computer-use	~/.local/bin/codewhale-computer-use, ~/.local/share/applications/net.codewhale.computer-use.desktop, hicolor icons; --login: ~/.config/autostart/
Windows	%LOCALAPPDATA%\Programs\Codewhale Computer Use	Start Menu shortcut with icon; --login: Startup-folder shortcut

macOS permission recovery

The build signs the complete app bundle, and installation signs again after pinning Node, then verifies the installed signature. Signing only the launcher leaves an invalid app identity: macOS can show an enabled switch while refusing the grant and repeatedly prompting for Accessibility.

If upgrading from that broken package, remove the old Codewhale Computer Use entry from Accessibility, add ~/Applications/Codewhale Computer Use.app, and enable it. Refresh its Screen & System Audio Recording grant and restart the helper. The probe must report Accessibility granted and screen capture ok; via: "app" alone is not a pass. Authentication in System Settings is handled by the user. Never modify the TCC database.

Builds use CODEWHALE_CU_SIGN_IDENTITY when configured, otherwise an available Developer ID or Apple Development identity, and fall back to ad-hoc signing. Ad-hoc updates can require a new grant. Installation preserves the selected identity and verifies the finished bundle. Building the native macOS helper requires Xcode Command Line Tools on a Mac; the separate Computer Use app download includes the compiled helper. Do not edit installed resources without re-signing.

Background control and visible cursor on macOS

Use open_application with activate: false to select the input destination without bringing it forward. Keyboard events go to that process and semantic actions use Accessibility — both are quiet: no cursor movement, no activation. Unqualified get_app_state, list_windows, and screenshot follow that app, including behind other windows. Under the 0.13.0 helper, explicit display/region captures require host approval and are refused when an app exclusion could expose private content. Prefer an app-scoped capture. Text insertion prefers writable accessibility selection over process key events. For an authorized workflow that requires keyboard focus, explicitly select activate: true; receipts then say keyboard_delivery: "foreground-guarded". Keystrokes and raw mouse gestures stop if another app takes focus; gestures never reactivate it. Select activate: false again to return to process-bound delivery. In either mode, verify the application result: a successful dispatch alone does not prove the app handled it.

Coordinate left_click first resolves the bound application's accessibility control and uses accessibility (strategy: "a11y"). Text fields focus directly, rows can select, and menu items can use their advertised pick action. Right-click uses an advertised context-menu action. Background scrolling uses the selected scrollbar: native increments when advertised, otherwise normalized 5% steps, with the actual unit and value change in the receipt. Unsupported gestures, including raw double/triple/middle click, drag and hover, remain unavailable in background mode. There is no automatic foreground fallback.

When the authorized task needs focus in an allowed app, select activate: true for foreground control without a second approval prompt. Raw clicks, hover, drag and wheel then go to the bound app's window as window-routed event records (strategy: "window-record", pointer_moved: false) — the agent has its own pointer, drawn in the preview, and the user's cursor is never moved, warped or held on macOS. There is no real-cursor fallback: a helper that cannot resolve the window route refuses bg_dispatch_unavailable. pointer down/move/up buffer a drag and deliver it to the window on up. Return to activate: false when the foreground step ends.

activate: false is the default on every platform, not just macOS: Windows launches the app minimized and Linux hands focus back to the previous window after launch. Raw input on Windows and Linux is still shared-surface by nature — background there means the launch does not steal focus, not that input becomes background-safe.

The binding receipt exposes input_scope, shared_pointer: false, pointer_route: "window-record" and isolated_desktop: false. The preview title distinguishes background app control from shared-desktop control. It is a view of the app, not a sandbox.

The 0.13.0 candidate uses a compact Codewhale controller naming the app, with a direct Stop computer use button and Pause in its menu. Stop ends all current computer sessions. Idle shows Ready; Settings holds remembered access, previous connections and the optional background diagnostic. It is Codewhale's own UI, not an assertion that macOS shows an identical system indicator. It does not capture the screen. The screenshot preview is off by default.

Only when the user asks to watch, enable preview with enabled: true to show a small, nonactivating window containing the controlled app and a cyan cursor labeled Codewhale. The preview refreshes on a timer while a session is bound (CODEWHALE_CU_PREVIEW_REFRESH_MS, default 1000; 0 disables), so it behaves like a live view of the app instead of a frozen still; its cursor is separate from the hardware pointer. Use the Close button, Close preview in the Computer Use menu bar, or enabled: false to stop watching. The panel also hides after 60 seconds without session activity or when that session closes; a session that disappears stops renewing the panel, which then expires.

Consent and shared-machine manners on the local computer

The 0.13.0 Mac helper owns local admission. A native decision grants this Codewhale installation permission to use the Mac until the person revokes it in Computer Use. The agent can then choose apps needed for the task without per-app, per-task or per-model prompts. Other hosts without a persistent key receive connection-scoped approval. Discovery and consent {action:"status"} do not themselves ask for access. The latter reports the helper's actual trust and exclusions. An installed older plugin may still use the legacy per-app ledger until it is updated.

Codewhale supplies its stable secret on the plugin's private stdin channel. The plugin derives an Ed25519 identity and signs a fresh helper lease challenge; the native helper stores only the approved public key and exclusions in Keychain. This proves possession of the same installation key, not the vendor identity or code signature of a running app. It is not an OS sandbox: same-user code able to read Codewhale's secret store can impersonate that installation. Details and qualification limits are in Limitations.

App and foreground exclusions still win. On the current helper, app consent deny or revoke narrows access; it cannot clear a native exclusion. Whole-host revocation is a human control in Computer Use. The agent must not retry a denial under another name, session or tool. macOS Accessibility, Screen Recording and Automation permissions remain separate OS decisions.

Whole-screen captures can show unrelated apps; use scoped observations and captures for the task. A host with app exclusions cannot take an unscoped whole-screen capture or use a browser route that cannot enforce those exclusions. Trust does not start a recording or preview. Spawned task computers and remote computers retain their existing transport rules.

app_script needs a fresh approval for the exact script; the helper enforces its single-use token and app exclusions. Irreversible-action confirmations and computer registration or spawning still need a host-attested decision or MCP elicitation. Without a supported decision path the call refuses consent_needs_user.

For macOS hosts without the helper, an interim osascript dialog is available only with CODEWHALE_CU_NATIVE_PROMPT=1; it is off by default. An installed but disconnected helper is never bypassed. While a helper prompt is open, other input and observation are refused with consent_pending; Stop cancels it. Prompts are rate-limited, and declining or timing out does not authorize a retry. See docs/LIMITATIONS.md for the decision paths and their verification limits.

In explicitly selected foreground mode for an approved app, whenever a shared surface is taken — a front lease for window-record input, a real-pointer gesture, foreground keys, or an activation — the helper waits for a gap in the user's hardware input first (bounded, ~450 ms gap within a 2.5 s window by default; CODEWHALE_CU_YIELD_GAP_MS / CODEWHALE_CU_YIELD_WAIT_MS). If no quiet window arrives, the call refuses user_busy before sending input. Successful receipts report the wait as yield_ms. Turn-taking, not a lock: user_input_during_lease still reports input that arrived mid-action.

Browser control (CDP)

browser drives a Chromium-family browser over the Chrome DevTools protocol in a self-owned profile under the state dir — the person's own browser is never attached to, typed into, or closed. start {url?} launches (or reuses) the instance and binds this session's own tab; navigate, click (a CSS selector's box center, or a page-viewport point), type (optional selector + enter), screenshot, status, and stop follow. Page screenshots return as inline images like screen captures do. stop closes this session's tab; the shared browser goes down when no tabs remain. Coordinates in this family are page-viewport pixels (space: "page-viewport") — never screen points. Node 22+ is required for the WebSocket transport; older runtimes refuse with unsupported_runtime.

This is background control of a local app, not an isolated desktop. Some apps, system dialogs, and workflows may still require foreground interaction. The opt-in node scripts/verify-macos.mjs test exercises a uniquely named TextEdit fixture (closed automatically afterward; add --preview to test the overlay), Unicode input, selection, screenshots and foreground preservation through a fresh MCP connection. Recording is separately opt-in with --recording. It creates local receipts and is separate from the automated unit suite.

Logs: ~/Library/Logs/Codewhale Computer Use/app.log (macOS), ~/.local/state/codewhale-computer-use/app.log (Linux), %LOCALAPPDATA%\Codewhale Computer Use\app.log (Windows). npm run remove:app reverses the install. Every bundle carries a complete runtime copy of the plugin, so hosts can target the installed path and survive deleting this checkout.

Session ownership

Each MCP connection has independent computer selection, application binding, rasters, accessibility observations and held-input ownership. Local actions are serialized by the permission-owning app; stopping or cancelling one session releases its own held input. Normal client shutdown closes the app session; losing the client connection also cancels and releases its input. On macOS, the native input helper also releases its press when its owning process disappears, including in direct mode. On macOS, session exit stops a recording started by that session; cancelling an ordinary request leaves an explicitly started recording running until stopped or the session exits.

Session protocol 2 (introduced in 0.2.1) is required: upgrade the helper and restart existing MCP connections together. An old client or helper is refused with an upgrade error instead of sharing another client's input state.

Frontier ability set

The advertised surface is merged for context economy — click, pointer, clipboard, recording, computer, browser, with key {duration} covering holds; list_sessions, kill_app, set_window_frame and trajectory ride alongside as themselves, and list_apps {installed:true} returns the installed catalog rather than running processes.

Capability grants

CODEWHALE_CU_GRANT narrows a server process at launch: read-only, or a comma list of tool names (a merged name expands to its whole action set). tools/list advertises only granted tools, ungranted calls fail as not_granted before validation, and the app daemon enforces the same set on the session lease so a narrowed server cannot smuggle one through. Cleanup and stop_computer_control are never blocked; request_access reports the grant. The per-action wire names (left_click, read_clipboard, recording_start, computer_list, hold_key, …) remain callable as aliases, so pinned hosts and receipts keep working; tools/list advertises the merged set only.

• Observe & resolve — list_apps (regular running apps by default; all:true for helpers, installed:true for the launchable app catalog), list_windows, list_displays, switch_display, get_app_state (accessibility/UIA/uitest tree with element indices + state_id), screenshot (display/region, raster-bound coordinates), zoom (close-up crop of the last raster), cursor_position, open_application (exact-name rule), request_access (fail-closed permission/capability probe), wait_for (poll the accessibility tree until a query/role appears or disappears, then act on the fresh state_id).
• Pointer — click (left/double/triple/right/middle through button/clicks), pointer (move/down/up), left_click_drag, scroll (4 directions).
• Keyboard & text — type (unicode), key (chords, repeat, or duration to hold), set_value (semantic, background-safe), select_text, perform_action (element's own actions: AXPress / UIA Invoke / AT-SPI / uitest), invoke_menu (menu items by title path; accessibility only — no key events or focus lease; window-targeted items may need a key window).
• Scripting — app_script (macOS, local computer): AppleScript or JXA through osascript into apps that ship a scripting dictionary — the programmatic interface to prefer over clicking wherever one exists. Returns stdout as result; refusals are typed (script_error, script_timeout, automation_denied for a declined/missing Automation consent). Refused on ssh/docker/hdc computers: a remote channel stays computer-use only, never a shell.
• Guidance & policy — the operating skill ships as MCP resources (skills/list, skills/get, resources/read of skill://codewhale-cu/…, sha256 manifest) and as the skills/computer-use/ pack in this repo; every tool advertises MCP annotations (readOnly / destructive / idempotent / openWorld) for host approval and sandbox policy.
• Recording — recording (start/stop/status/list; see below).
• — (list/switch/register with ssh agent auto-push/spawn/remove). provisions a task-owned disposable Linux desktop container: it is registered , becomes active, takes every tool unchanged through the same agent protocol as ssh, and is destroyed by or when the MCP session ends, and stops after its lifetime (six hours by default). A computer is an execution environment — prefer a spawned desktop for work that does not need the user's own session, and keep for the tasks that do.

Requirements

Tools and permissions are probed at call time; request_access reports what is missing and every capability fails closed naming the missing tool or permission — it never guesses and never half-acts.

• macOS — Accessibility + Screen Recording for the app (or, in direct mode, for the terminal that hosts the server). python3+pyobjc or cliclick improves cursor reads.
• Windows — PowerShell (built in). Recording is unavailable pending session-owned recorder cleanup.
• Linux — X11: xdotool, wmctrl, scrot or imagemagick, xclip, and ffmpeg for recording (x11grab); Python 3 and xmodmap for non-ASCII typing; python3-pyatspi for the accessibility tree. Wayland: grim screenshots, wl-clipboard and AT-SPI. Keyboard and pointer input work on sway and Hyprland (python3 and libxkbcommon, both present on those desktops), where the compositor names the app that would receive the input. GNOME, KDE and other compositors do not, so input is refused there. Recording is unavailable on Wayland.
• HarmonyOS — hdc on PATH with the device connected (hdc list targets); ffmpeg on the host for snapshot-series recordings.
• Spawned computers — a reachable docker daemon (docker version). computer spawn builds the plugin's Linux desktop image from docker/Dockerfile on first use; no sshd, keys or host mounts are involved — the agent rides docker exec and the container boundary is the isolation.

How the four platforms map

Ability	macOS	Windows	Linux	HarmonyOS
Accessibility tree	Native Accessibility API	UIAutomation	AT-SPI (pyatspi)	uitest dumpLayout
Raw input	Native CGEvent to the selected process	user32 SendInput/mouse_event (PowerShell)	xdotool (X11); virtual keyboard and pointer on sway/Hyprland; refused on other Wayland compositors	uitest uiInput
Screenshots	screencapture	.NET CopyFromScreen	scrot/import (X11), grim (Wayland)	snapshot_display
Recording	ScreenCaptureKit → .mov (no recorder overlay)	unavailable pending owned cleanup	ffmpeg x11grab → .mp4 (X11); unavailable on Wayland	snapshot-series + ffmpeg mux
Clipboard	pbcopy/pbpaste	Get/Set-Clipboard	xclip/xsel, wl-clipboard	fail-closed (not exposed by hdc)

Remote computers (ssh)

computer_register { "computer": "winbox", "transport": "ssh", "host": "winbox.lan", "user": "me" }

Registration pushes the self-contained agent (agent.mjs + src/) to ~/.codewhale-cu/agent/ on the remote over scp, probes the remote platform through it, and pins the result. Remote calls run node agent.mjs <base64 json> — one JSON receipt line back. Only an allow-listed tool set executes remotely; arguments travel as data, never as shell. Requires publickey ssh (BatchMode) and Node ≥ 20 on the remote.

HarmonyOS computers

computer_register { "computer": "pad", "transport": "hdc" }

Drives the device over hdc shell uitest ... and snapshot_display. Element targets come from dumpLayout; input is touch-synthesis (click / swipe / inputText / keyEvent). Recording is honestly labeled snapshot-series (frame captures muxed on stop) because HarmonyOS exposes no CLI screen recorder.

Verification status

0.13.0 candidate. A signed build passed native human approval, Unicode input/click/scroll/app capture, reconnect through a new MCP process after helper restart, and human revocation on one maintainer Mac. All 115 observer samples had no pointer or foreground changes. The subsequently simplified native UI and release packaging still need qualification. Focused source checks alone do not qualify an installed app. Final counts and release receipts belong in the release notes and release checklist.

Earlier source-suite and live receipts below cover their named versions. Protocol, routing, session and injected-runner tests perform no native input and do not qualify a distributed app.

Live-verified: macOS 0.8.0 acceptance (one maintainer Mac, arm64, 2026-09-17). A muse-driven pass over the installed 0.8.0 plugin plus fixes: 36 advertised tools with hidden aliases callable; TextEdit invoke_menu open/close with front_lease/front_restored honesty on every key receipt; read-only grant narrowing (19 tools, not_granted, grant on refusal receipts too); a 3-turn trajectory with a refusal that replay stopped at; 143 installed apps with running flags; TextEdit frames with verified geometry and Calculator's fixed-size refusal via ax_errors; browser start/navigate/click/type/screenshot/stop in a self-owned profile with the user's Chrome untouched; session list, live preview refresh/hide, and verified kill_app; malformed calls failing as bad_args/bad_target (never TypeError); element_stale naming state and app; the kill switch failing closed; and the real pointer provably unmoved across background clicks. Tracked as Codewhale CU SHA-6642.

Native macOS 0.4.0 (one maintainer Mac, arm64). The 0.4.0 universal app was Developer ID signed and Apple-notarized from commit 249ae77, then installed over the notarized 0.3.1 app on the maintainer Mac; the packaging facts are in docs/releases/0.4.0.json. The AX primitives it adds (Return from type, filtered get_app_state, focus, get_value, strategy:"app") are covered by the source suite; their native qualification on an installed build is still open.

Native macOS 0.3.1 (one maintainer Mac, arm64). An earlier signed 0.3.1 candidate passed the menu-bar owner crash and reopen check with isolated state. The updater's apply step replaced an installed notarized 0.3.0 app with that candidate: the previous bundle was retained for rollback, the helper restarted with controls stopped, and all 33 runtime files plus the 3 native executables in the installed bundle matched the build. The open qualification gates, and the build, signing and notarization facts for the public 0.3.1 archive, are in CHANGELOG.md and docs/releases/0.3.1.json.

Live-verified during development: macOS (arm64, single Retina display, macOS 26.1). Each of the 27 fixture workflows has a five-trial passing run. The broad 26-task run passed 129/130 trials; its dynamic-page failure was a fixture clock race, corrected and repeated 5/5. The repaired file-picker flow also passed 5/5 separately. The matrix retains the broad failure and both focused runs; these are 0.2.1 development receipts, not full Codex parity or final release qualification. Shared-desktop pointer displacement is measured and sometimes nonzero. OS permissions survived the signed 0.2.1 update. Commit identifiers inside the matrix and result files refer to the private pre-publication history, not to commits in this repository.

Text and vision use the same actions. App observations default to a text summary containing controls, values, actions and layout. detail:"full" exposes nested menus and tree structure. On macOS, include_ocr:true adds on-device recognition of visible text with confidence and coordinate targets, without a vision model or remote service. The default does not capture an image. OCR was verified on a generated image and the actual Codewhale app; it does not interpret unlabeled icons, charts or other graphical meaning. Screenshots and zoom remain available to models that support images.

Linux X11 (experimental, from source). Verified on 2026-10-11 on virtual X11 desktops (Xvfb with openbox, and a full XFCE session) and in the Docker test image, at repeats 5 over the 27-task suite. GNOME and KDE sessions have not been tested.

These 2026-10-11 results are historical and do not establish the current helper route. The 0.13.0 source candidate has a GTK/polkit approval controller; its isolated Linux checks do not qualify other desktops. Direct mode remains available for development when explicitly selected by the user and does not support held gestures.

route	demonstrated	not demonstrated
isolated Xvfb, direct	24/27	the two drag tasks (need the desktop helper); native.modal_dialog 0/5
isolated Xvfb, desktop helper	25/27	control.permission_denied (a direct-route check); native.modal_dialog 0/5
shared display, direct	23/27	the two drag tasks; browser.unicode_type 4/5; native.modal_dialog 0/5
shared display, desktop helper	25/27	control.permission_denied; native.modal_dialog 0/5
Docker image, direct	23/27	the two drag tasks; browser.unicode_type 2/5; native.modal_dialog 0/5
Docker image, desktop helper	23/27	browser.unicode_type 4/5; browser.modifiers 4/5; control.permission_denied; native.modal_dialog 0/5
XFCE session (xfce4-session, xfwm4, panel) on Xvfb, direct	23/27	the two drag tasks; browser.unicode_type 4/5; native.modal_dialog 0/5
XFCE session, desktop helper	24/27	browser.unicode_type 4/5; control.permission_denied; native.modal_dialog 0/5
Wayland: headless sway 1.9 (XWayland for Tk), direct	21/27	the two drag tasks; browser.upload 0/5 (GTK's header-bar file chooser puts Open where the task's X11-layout click does not reach); browser.unicode_type 0/5 (Chromium on Wayland drops 🐳); native.unicode_type 4/5; native.modal_dialog 0/5

Known limits: on X11, input to a window is refused while a visible dialog explicitly identifies that window as its parent. Interact with the dialog or close it before returning to the parent. This conservative rule also blocks some non-modal dialogs, and cannot identify every toolkit dialog or eliminate window changes between the check and input. On Wayland a toolkit-modal dialog can still let typed text reach its parent, so click into the dialog first. X11 non-ASCII typing holds a temporary key mapping while sending input, but delivery is not guaranteed; read it back with get_value. On Wayland, input goes only to the app that sway or Hyprland names as its receiver: the focused window's process for keys, and the window under the point for the pointer. It is refused where that cannot be told: overlapping floating windows of different apps, overlapping XWayland windows of different apps, and pointer input at the current pointer position on sway, which does not report it. GNOME, KDE and other Wayland compositors name no receiver, so input there is refused, leaving screenshots (wlroots only), the clipboard and the accessibility tree. Hyprland is unit-tested only. Not tested: GNOME or KDE sessions, a real sway login session, real XWayland on a desktop, multiple monitors, HiDPI, SSH remote. Receipts are in parity/results/; docs/PARITY_MATRIX.md and docs/LIMITATIONS.md have the detail. The Linux helper requires the system approval setup above before it can perform held gestures.

Background input has a native AppKit verification harness with an independent foreground/cursor observer: node scripts/verify-background-macos.mjs. Older per-family receipts do not establish background isolation — details in docs/LIMITATIONS.md.

Not live-verified: macOS non-Retina and mixed-DPI, Windows, Wayland beyond headless sway, HarmonyOS, SSH remote — implemented, no device receipts. A same-document native editing comparison completed 5/5 trials with each of Codewhale and Codex; it covers one workflow, not the full task suite. A separate public-browser navigation comparison also completed 5/5 trials on each surface, using Codewhale 0.2.1. Known behavioral limitations and the release gating checklist are in docs/LIMITATIONS.md and docs/RELEASE_CHECKLIST.md; how to run or extend the suite is in docs/PARITY.md.

Layout

mcp/server.mjs        MCP stdio server (JSON-RPC 2.0), tool dispatch, receipts
src/tools.mjs         tool schemas — single source of truth for tools/list
src/backends/         darwin / win32 / linux / harmonyos
src/transport.mjs     app socket · local · ssh · hdc executors
src/app-socket.mjs    app naming, socket protocol client, launch-on-demand
src/app-handler.mjs   allow-listed request handler shared by app + ssh agent
app/daemon.mjs        the desktop app process
app/macos/launcher.c  native bundle executable (keeps TCC attribution on the app)
agent.mjs             ssh remote agent (one-shot, or `--serve` persistent session)
assets/               icon source + generated .png/.icns/.ico/hicolor, prebuilt mac launcher
scripts/              build-icons · build-app · install-app · smoke
commands/, skills/    Agent Plugins v1 command + skills for hosts that read them

Development

npm test              # unit + protocol + app tests (no GUI input performed)
npm run smoke         # live end-to-end against this machine (isolated state dirs)
npm run build:icons   # regenerate assets/ from assets/icon-source.png
npm run build:app     # bundles into dist/ (compiles the mac launcher when clang is present)

Proven levels are separated: local live (this Mac: darwin) > mocked transport (ssh protocol, harmony backend logic) > code-complete (win32/linux paths, implemented to their documented tool interfaces but only verifiable on those platforms).

This repository supplies the independently installed Computer Use plugin and native helper. Codewhale installs and updates the plugin through Connectors from a stable published release; the helper has its own verified app updater. The 0.13.0 candidate plans a portable computer-use.tar.gz release asset for the existing plugin installer. Changing this checkout updates neither installed component. Plugin and helper updates do not require rebuilding Engine.

Support and contributing

Use GitHub issues for bugs and questions. Include the plugin version, macOS version, the failed action and the error text, with private app contents and credentials removed from any log excerpt. docs/TROUBLESHOOTING.md covers the common setup problems first. Contribution expectations are in CONTRIBUTING.md; vulnerability reporting and the security model are in SECURITY.md. This is a beta maintained on a best-effort basis; there is no support commitment or response-time promise.

License

MIT — see LICENSE.