Handy AI
An Android app that does tasks on your phone for you: say or type what you want, and it uses Claude to look at the screen and tap, type and swipe through your apps until the job is done. It is built first for blind, low-vision and motor-impaired users, and works for anyone who wants hands-free control of any app.
Phones assume you can see the screen and hit small targets. A screen reader tells you what is there, but a five-screen task like "message Sarah that I'm running late" is still dozens of precise gestures. Handy AI turns the sentence into those gestures, can read every step aloud, and stops when you tell it to.
Watch the 60-second demo on YouTube
Quick start
You need an Android 11+ phone and an Anthropic API key (starts with sk-ant-; Anthropic bills you per use).
- Install from Google Play, or download the signed APK from gethandyai.app (about 64 MB) and open it. Your browser will ask you to allow installs from it once.
- Add your key. Open Handy AI, paste the key into the bar at the bottom and tap Save key. If pasting a long key is hard (for example with a screen reader), use Settings > Set up via computer instead: the phone shows a 6-character code (and speaks it if voice is on), and you paste the key on gethandyai.app/setup on your computer.
- Give a task. Type or tap the mic, for example "Open the camera and take a picture".
- Turn on the accessibility service when asked (first task only): Android Settings > Accessibility > Handy AI > On (on some phones Handy AI is under Installed apps or Downloaded apps). Come back and the task starts by itself.
- If the switch is greyed out with "Restricted setting" (Android 13+ with a downloaded APK), open Settings > Apps > Handy AI > ⋮ > Allow restricted settings and try again.
- Watch it work. Each step is shown in the chat (and spoken if voice is on). Tap the stop button to cancel at any time.
The APK on the website can lag behind main. At the time of writing it is 1.5.5 (Claude Opus 4.7); main is 1.5.6 (Claude Opus 5.5). Build from source for the latest.
What a run looks like
This is the run in the screenshot above, copied from the app's chat log (v1.5.1 on a Pixel 10). Indented lines are a status row and the tool calls Claude made:
You: take a picture with the phone camera
Screen: 1080x2424
Claude: I'll go to the home screen and open the camera app to take a picture.
key({"action":"home"})
Claude: I can see the Camera app icon. I'll tap it.
tap({"x":336,"y":2035})
Claude: The camera is open in Photo mode. I'll tap the shutter button to take the picture.
tap({"x":540,"y":1810})
Claude: The picture has been taken - I can see the new thumbnail in the bottom-left preview
now showing the image of the office with the chair.
finish({"success":true,"summary":"Opened the Camera app and tapped the shutter button to take a picture. The new photo appears in the thumbnail preview."})
Claude: Opened the Camera app and tapped the shutter button to take a picture. The new photo
appears in the thumbnail preview.
More things to try:
| Say or type | Notes |
|---|
| "Message Sarah on WhatsApp that I'm running late." | If voice is on, you get a spoken warning and a pause before it taps Send |
| "Search Google Maps for sushi near me." | |
| "Read me the screen." | Claude calls the on-device OCR tool and the text is read aloud |
| "What did you do?" | Answered on the phone from this chat's log, no API call |
| "What can you do?" | Answered on the phone with example tasks, no API call |
Features
- Voice in and out. Android speech recognition for input; Android text-to-speech for replies and step narration (Off / Brief / Detailed). Voice mode can follow TalkBack automatically.
- Destructive-action warning. With voice on, if Claude says it is about to send, buy, delete, post (and similar), the app speaks a warning and waits 1.5 s so you can hit Stop.
- Easy Mode. The whole app becomes one large mic button.
- Named aliases. Teach it that "Mom" means "contact Sarah Cohen" or "my usual" means a specific order.
- Screen reading.
read_screen_text / read_text_at run Google ML Kit OCR on the device.
- Accessibility shortcut. The floating accessibility button opens Handy AI from any app.
- Local chat history with multiple sessions in a side drawer.
Configuration
All settings are in the app (gear icon). Nothing is configured with environment variables at runtime.
| Setting | Default | What it does |
|---|
| Anthropic API key | none | Required. Saved from the key bar or Settings; must start with sk-ant- |
| Voice mode | Auto | Off, On, or Auto (speak only while a screen reader such as TalkBack is on) |
| Narration | Brief | How much of each step is spoken: Off, Brief, Detailed |
| Voice-warn before destructive actions | On | Spoken warning and 1.5 s pause; needs voice to be on |
| Vibration feedback | On | Patterns for started, done, needs attention, error |
| Easy Mode | Off | Single mic button UI |
| Named aliases | none | Added to the prompt so Claude can resolve them |
| High-contrast text | Off | See Limitations |
Agent constants live in code:
| Constant | Value | File |
|---|
| Model | claude-opus-5-5 | AgentLoop.kt (MODEL, the only place it is set) |
Effort / max_tokens per step | low / 8192 | AnthropicClient.kt, AgentLoop.kt |
| Max steps per task | 40 | AgentLoop.kt |
| Screenshots kept in the conversation | the task's first one plus the 4 most recent; older ones become a text note | AgentLoop.kt |
| Settle delay after each action | 700 ms | AgentLoop.kt |
| Retries | up to 4 attempts on HTTP 408, 429, 500, 502-504, 529 and network errors | AnthropicClient.kt |
Permissions and privacy
Permissions
| Permission | Why |
|---|
INTERNET | Calls the Anthropic API |
VIBRATE | Haptic feedback |
| Accessibility service (you turn it on in Android settings) | Take screenshots, perform taps and swipes, set text in the focused field and press Enter, press Back / Home / Recents, show the accessibility shortcut button |
The accessibility service only acts while a task you started is running. It does not process accessibility events (the handler is empty). Its config also declares window-content access, which is used to find the focused text field.
What leaves your phone
| When | What | Sent to |
|---|
| Every step of a running task | A JPEG screenshot of the whole screen (notifications and anything else visible included), your task text, your named aliases, earlier steps of this task, action results, and any text the OCR tools read | api.anthropic.com, directly from the phone with your key, under Anthropic's API terms |
| Only if you use Set up via computer | Your API key, typed on your computer | Relayed to the phone through the author's server (dashob.com). The setup page says it is held in memory for at most 10 minutes and deleted when the phone collects it |
| Only if you pair in Settings > Cloudbot pairing | A device token, a random device ID and your phone's model name | The author's hub server, over a WebSocket |
Apart from the optional rows above, nothing is sent while no task is running. OCR runs on the device with Google ML Kit, which may send Google usage metrics under its own terms. Crash reporting (Sentry) is compiled in but only starts when a DSN is set at build time; the published APK has none.
What stays on your phone
- The API key, in
EncryptedSharedPreferences (AES-256, key in the Android Keystore). If the Keystore is unusable on a device, it falls back to plain app-private storage.
- Chat history (your tasks, Claude's replies, the tool calls including typed text), in app-private storage. Clear it by clearing the app's data.
- Android backup is disabled, so none of this is copied to Google Drive.
Safety. Claude is instructed not to enter passwords or payment details, and not to delete, buy, uninstall or send money unless you asked for that exact thing. That is an instruction to the model, not a sandbox: the service can tap anything you can. Watch what it does and use Stop.
How it works
you: "take a picture"
|
MainActivity / EasyModeActivity
| startAgent(key, task)
AgentAccessibilityService ---- TaskRunner (one task at a time, Stop = cancel)
|
AgentLoop, up to 40 steps:
1. takeScreenshot() (Android 11+ accessibility API)
2. POST /v1/messages (AnthropicClient: screenshot + history + tools)
3. Claude replies with a tool call (AgentProtocol parses it, AgentActions validates it)
4. run it: dispatchGesture / ACTION_SET_TEXT / global action / OCR
5. wait 700 ms, take a new screenshot, send it back as the tool result
... until Claude calls finish, Stop is pressed, or the step limit is hit
Tools Claude can call: tap, long_press, swipe, type_text, clear_text, key (back, home, recents, ime_enter), wait (up to 3 s), read_screen_text, read_text_at, finish. Coordinates are screen pixels; the screen size is given in the system prompt.
| Path | Role |
|---|
app/src/main/java/com/claudeagent/phone/AgentLoop.kt | The screenshot, Claude, action loop |
AgentProtocol.kt, AgentActions.kt, AgentTools.kt | Response parsing, history trimming, system prompt; tool input validation; tool schemas sent to Claude (pure Kotlin, unit tested) |
AnthropicClient.kt | OkHttp client for the Messages API with retries and immediate cancel |
AgentAccessibilityService.kt, TaskRunner.kt | Screenshots and gestures; runs one agent task at a time |
AccessibilityHelper.kt, ScreenOcr.kt | Text-to-speech, haptics, the destructive-action heuristic; ML Kit OCR |
MainActivity.kt, EasyModeActivity.kt, SettingsActivity.kt, KeySetupActivity.kt | UI |
LocalIntents.kt, NameAliases.kt, ChatStore.kt, ApiKeyStore.kt | On-device answers, aliases, chat history, key storage |
HubClient.kt, HubConnection.kt, HubPairActivity.kt | Optional pairing with the author's agent hub |
OnboardingActivity.kt, SupabaseAuth.kt, AuthStore.kt, EntitlementClient.kt, LemonSqueezyClient.kt, BillingConfig.kt | A trial and subscription mode that is currently switched off (its backend was deleted; see supabase/REBUILD.md) |
PanelClient.kt, PanelPairActivity.kt | Pairing with a retired dashboard; kept only so old pairings can be cleared |
docs/ | The gethandyai.app site, including the APK and the computer key-setup page |
playstore/ | Store listing assets, screenshots and demo videos |
Limitations
- Android 11 or newer (the accessibility screenshot API).
- Cost and speed. Every step is one Claude request with a full-screen image, plus a 700 ms settle delay. Longer tasks cost more; check usage in the Anthropic console.
- 40 steps per task at most.
- Some apps hide their content from screenshots (common in banking and password screens); Claude may see a black screen there.
- Typing can fail in web fields that do not accept accessibility text input. Claude can sometimes work around it by tapping the on-screen keyboard.
- Android may switch the service off after an app update or under memory pressure. Turn it back on in Accessibility settings.
- The destructive-action warning is a keyword check on Claude's last message, not a guarantee.
- High-contrast text has little visible effect today: most layouts use fixed colors instead of theme colors.
- No in-app way to delete a single chat yet; clear the app's data to remove all history.
- Bring your own key only. The trial and subscription code paths are dormant.
Build from source
Requirements: JDK 17 and the Android SDK (platform 35). Android Studio provides both.
git clone https://github.com/fainir/handy-ai.git
cd handy-ai
echo "sdk.dir=$HOME/Library/Android/sdk" > local.properties # your SDK path; or export ANDROID_HOME
./gradlew assembleDebug
adb install app/build/outputs/apk/debug/app-debug.apk
The debug build is signed with your local debug key, so uninstall a Play or website copy first (same package name, different signature). No secrets are needed: the optional values in local.properties.example (Supabase, Google sign-in, Sentry) fall back to placeholders and only affect the dormant features.
Release builds read signing details from a git-ignored keystore.properties (storeFile, storePassword, keyAlias, keyPassword), then ./gradlew assembleRelease bundleRelease.
Development
./gradlew testDebugUnitTest # JVM unit tests: no device, no network, no API calls
./gradlew lintDebug
CI (.github/workflows/ci.yml) runs assembleDebug testDebugUnitTest lintDebug on every push and pull request. The unit tests cover response parsing, tool input validation, screenshot-history trimming, the system prompt, local intents, error messages, task cancellation and the hub WebSocket (against a local mock server).
License
MIT
- agent-harness - make Claude Code or Codex act as a product team, a CEO or a portfolio manager
- frugal - your agent finds the cheaper honest way before it pays for anything
- flipback - an automatic
/goal for every turn: sends Claude Code back when it stops early
- recall - your Claude Code memory in every agent, over MCP
- All projects
