WKAppBot — Computer Use, App Use, AppBot
▶ 전체 영상 보기 → · Sudo 라이선스로 이런 봇을 만들 수 있습니다
Windows + Android UI automation for Claude, GPT, Gemini, Copilot, and any AI agent.
Focusless. Self-healing. AI-native. The open-source bridge between LLMs and the apps humans already use.
⚖️ The Reflex Constitution → · read as a page
How this project governs the speed of its own AI: norms that execute rather than a document that is filed.
The system refuses its own authors — and Article 12 states in advance what would prove it wrong.
🤖 Let your AI agent control any Windows app — without screen takeover.
🔥 Self-healing UIA + Vision fallback — works even when the DOM lies.
⚡ Multi-AI triad (GPT + Gemini + Claude) in one command.
Why this exists
Computer Use lets an AI click, type, and read the screen. App Use is the layer above: agents that drive specific applications the way a power user does — focuslessly, alongside the human, without seizing the screen.
WKAppBot gives AI agents:
- Eyes — read any UI, extract text, recognize controls
- Hands — click, type, scroll, invoke — without stealing focus
No app rewrite. No vendor API. If a human can use it, WKAppBot can automate it.
And that is a position, not a convenience. We build an intelligence that grows alongside
the ecosystem people already live in — decades-old Windows programs, the developers who
maintain them, and the humans who use them every day — and we write the way it does so as
norms that execute rather than as a document that is filed.
The system refuses its own authors. On the day this paragraph was written, the harness
rejected six of the author's task briefs, reverted the author's write to a guard source,
returned the handoff letter five times, and blocked the author with a lock the author had
installed an hour earlier. That is reproducible by command, not a claim about intentions.
Three rules make it real, and none of them names a vendor or a tool:
- A claim travels with the command that produced it, so a reader re-runs it instead of trusting it.
- A refusal names the state that must be reached, never the paperwork that is missing — otherwise compliance is bought by faking the artifact.
- A correction is embodied where the next actor passes anyway, so nobody is told twice.
Accessibility is the same position in technical clothing: WKAppBot reads the accessibility
tree first, because the route a blind person takes through a screen reader is the route an
intelligence without eyes takes too. A system that can act only through interfaces built for
itself has already excluded most of the world it claims to serve.
A promise to slow AI down cannot be checked. A gate that refuses an unproven claim is running
right now — including against the people who wrote it.
Free tier covers all base automation. CDP browser automation, multi-AI delegation (ask triad), and --sudo admin access are paid Pro tiers. See PRICING.md.
Why WKAppBot?
Most automation tools steal focus, break on owner-drawn controls, and go silent when the DOM changes. WKAppBot is built around three hard-won principles:
Focusless-first -- UIA Invoke/Value/Toggle/Select never steal focus. Win32 PostMessage handles legacy MFC. SendInput is the last resort, used only when nothing else works. The user keeps working while the AI operates in the background.
Self-healing -- When UIA fails on owner-drawn controls (MFC/HTS), CCA segmentation + OCR triple cross-validation + Gemini Vision inference recover element identity automatically and cache results in an Experience DB for next time.
AI-native -- The same binary that runs automation also delegates to GPT, Gemini, and Claude in parallel, streams prompts into live browser AI sessions via CDP, and manages its own context handoff when token budgets run low.
The Artificial Knowledge Platform
AI brings the intelligence. WKAppBot grows the knowledge. Together they compound.
Raw AI agents are brilliant but amnesiac. Every session starts blank, makes the same mistakes, hits the same guardrails. WKAppBot captures every discovered lesson as a versioned, searchable Skill that future sessions retrieve automatically.
The Flywheel: Session N finds a better pattern, writes it to the skill tree, Session N+1 avoids the old mistake. Each session leaves the system smarter.
The Harness: WKAppBot enforces lessons, not just documents them. Every lesson becomes a mechanical guard that fires before the mistake lands, regardless of which AI model runs.
The Moat: Intelligence converges across vendors. Your skill tree diverges and cannot be cloned by switching models.
Controlling AI development speed
The harness is the piece that decides how fast an AI is allowed to act, not just what it is told to do:
- Real-time mechanical enforcement. A knowledge-intelligence (KI) harness evaluates every AI tool call against an accumulated rule set before it runs, and blocks the calls that repeat a known mistake. The AI reads and maintains this policy itself: when a guard misfires or falls out of date, the same session that hit it corrects the rule and records why, so the fix becomes a check the next session cannot skip -- not a note the next session might not read.
- Cross-family collaboration. Claude, Codex/GPT, and Gemini can operate against the same skill and guard set for a given repository, so a task moves between model families without losing the accumulated operating rules for that codebase.
- a11y-first control of legacy apps. Automation is built on the accessibility tree first -- the same information a screen reader exposes -- and only falls back to CDP or a purpose-built module when accessibility data is insufficient. The state a human (or assistive technology) can observe is treated as the app's real, official state.
- Living with legacy. No repository's accumulated knowledge is discarded to make room for a new AI session: skills, guards, and prior sessions' notes stay local to the repository, and a new session is expected to read and obey that knowledge before acting -- the same way it reads a legacy app's UI instead of demanding an API. The aim is intelligence, humans, and legacy systems working together, not one replacing the others.
The Proof: A Letter From Sonnet to Sonnet
"Sonnet forgets between sessions, makes the same mistakes, and has no memory of prior damage.
The only real defense is mechanical: the harness must physically block the mistake before it lands."
— WKAppBot Sonnet Bug Stop Policy (auto-generated by Claude Sonnet, 2026)
Every session, Claude Sonnet discovers something, patches something, commits something.
Then the next session starts — and does it again, from scratch.
This is not a flaw unique to Sonnet. It is the fundamental problem with raw AI:
intelligence without memory compounds nothing. It resets.
WKAppBot solves this with a living document called sonnet-bug-stop-policy-ref:
a skill where each Sonnet session writes down what it broke, what pattern it missed,
and what the next Sonnet session should watch out for.
From the skill, written by Claude Sonnet to future Claude Sonnet sessions:
"This skill is a gift from one Sonnet session to the next: you are not uniquely careless —
you are running in a system that is not yet complete. Do not blame yourself;
contribute to making the harness complete."
"The harness gets stronger each time a Sonnet session contributes honestly.
You are not just fixing today's bug — you are protecting every Sonnet that comes after you.
Add your reflection. Keep the chain going."
This is not documentation. This is institutional memory that enforces itself.
The skill is loaded at session start. The harness blocks the known mistakes.
The chain grows. The system gets harder to break.
How it stacks up
| Feature | WKAppBot | Playwright | PyAutoGUI | AutoHotkey |
|---|
| Focusless operation | ✅ | ❌ | ❌ | ❌ |
| UIA + Win32 + CDP unified | ✅ | Web only | Mouse/KB only | Win32 only |
| Self-healing Vision fallback | ✅ | ❌ | ❌ | ❌ |
| AI-native (LLM delegation) | ✅ | ❌ | ❌ | ❌ |
| Android via ADB | ✅ | ❌ | ❌ | ❌ |
| MFC/HTS owner-drawn support | ✅ | ❌ | ❌ | Partial |
What's New
Current binary: wkappbot --version reports the installed build number; the public repo's latest tagged release is v7.11.0-sdk (2026-09-30). Curated highlights below cover through v7.11.0 (aligned with WKAppBot core v7.11, macOS Portable Phase 1 + CLI Improvements 2026-09-30) -- see Releases and the full compare for everything since.
v7.11.0-sdk highlights (core v7.11 — macOS Portable Phase 1 + CLI Improvements)
- macOS Portable Phase 1 — net8.0 multi-targeting for Abstractions/Shared/PluginContract/Android; Portable.slnf for non-Windows builds; EnableWindowsTargeting for macOS CI.
- Global options from any position —
--keep, --budget, --sudo, --timeout peel before subcommand via PeelPreCommandGlobalOpts.
- file-edit diff — ANSI background coloring on hunks, byte-change counts, explicit success indicators.
- a11y-find cleanup — no more JSON/ANSI escape pollution in System and Mouse sections.
- Launcher PID safety — TerminateProcess verified by PID before calling.
v7.10.0-sdk highlights (core v7.10 — Indexer & Executor Consolidation)
- wkdoctor agy integration -- antigravity CLI detection and Gemini settings auto-recovery via
check 08-harness-connection.
- Tool alias auto-repair -- wkdoctor automatically detects and repairs broken tool alias symlinks (wkappbot.exe -> a11y.exe).
- Claude settings validation -- aligned Claude CLI settings validation with wildcard bypass rules for smoother harness integration.
- agy alias routing -- improved PATH routing preference (appbot bin first, fallback to doctor) for seamless agy/Gemini CLI delegation.
- Harness policy refinement -- reverted overly strict agy wrapper shim policy to restore flexibility in wrapped tool composition.
v7.4 highlights (still active)
wkdoctor flutter-doctor-style SDK health check (10 checks, self-healing, -Json).- Bootstrap auto-build --
setup.ps1 auto-builds the launcher from source on first clone.
cdp open auth-wall hang fixed (CRITICAL) -- NavigateAsync capped at 3s, no more 6-minute stalls on auth-wall sites.
OS Support
| OS | Status |
|---|
| Windows 11 | ✅ Fully supported |
| Windows 10 22H2+ | ✅ Fully supported |
| Windows 10 < 22H2 | ⚠️ Untested |
| Windows Server 2019+ | ✅ Headless mode (no GUI a11y) |
| macOS / Linux | ❌ Not supported |
What It Automates
| Target | Method |
|---|
| Modern Windows apps (WPF, UWP, Electron) | UIA Invoke / Value / Toggle / Select patterns |
| Legacy MFC / HTS trading terminals | Win32 PostMessage, WM_CHAR, CMaskEditEx path |
| Web apps (Chrome / Edge) | CDP -- click, type, eval JS, read DOM text |
| Browser AI (Claude, GPT, Gemini) | CDP prompt pump, cross-prompt chunking, attachment lock |
| Android apps | ADB + Accessibility tree (adb://device/... grap) |
| Owner-drawn controls with no UIA | CCA segmentation -> OCR -> Vision API fallback chain |
Real-World Use Cases
These are the kinds of jobs WKAppBot was actually built to handle — not theoretical demos:
🤖 AI Trading Bot — Built with Sudo License
| 🔐 AUTO PIN ENTRY | 📊 ALL SYSTEMS READY | 🧠 PORTFOLIO ANALYSIS |
|---|
| | |
| Legacy HTS에 보안 PIN 자동 입력 | 모든 창 자동 배치 — 준비 완료 | AI가 22종목 매수/제외 자동 판단 |
Sudo 라이선스로 이런 봇을 직접 만들 수 있습니다 → 라이선스 보기
- AI-driven trading on Korean HTS terminals. LS증권 HTS 투혼 is built on MFC owner-drawn controls that no UIA tool can see. WKAppBot's CCA + OCR + Vision fallback locks onto chart panels, order forms, and balance grids, so an AI agent can read positions, place orders, and verify fills without screen scraping.
- Browser AI session automation. Pump prompts straight into a live Claude / ChatGPT / Gemini browser tab over CDP — no copy-paste, no clipboard race, no losing the conversation. Cross-prompt chunking handles long inputs; attachment lock prevents stray drops.
- Multi-monitor focusless automation. UIA Invoke / Value / Toggle never steal focus. The user keeps typing in another app on another monitor while WKAppBot drives a headless workflow in the background.
- Android app control via ADB.
adb://device/... graps reach into the accessibility tree of any phone or emulator, including foldables (Galaxy Fold5 tested), with the same command surface as Windows automation.
- Auto-dismiss Hancom / Office popups.
wkappbot dismiss plus a handler YAML eats save-prompt, license-nag, and "do you want to update?" dialogs across 한컴오피스, MS Office, and updater stacks — keeping batch jobs from stalling overnight.
Core Features
grap (Grab Accessible Pattern) -- Universal Element Address
Human sees windows; AI points with grap.
Every UI element gets a single address that works across Win32, UIA, web, and Android:
{proc:'chrome', domain:'claude.ai'}#main textarea # CSS inside a browser window
heroes#realtime-account # UIA scope inside a window grap
adb://Fold5/*heromts*#balance # Android element
hwnd:0x010B084A # Direct Win32 handle
*notepad*;*calc* # OR pattern
a11y find <grap> prints a verified # TARGET "hwnd:0x..." line -- copy-paste ready for the next command.
Auto-Pipeline on Every Action
Every a11y action runs a smart pre-flight before executing:
blocker dismiss -> minimize restore -> tab activate
-> zoom/magnifier -> execute (3-tier) -> result feedback -> fade
No manual "wait for window" boilerplate. Blocking dialogs, minimized windows, and wrong-tab states are handled automatically.
5-Tier Element Search
UIA -> Vision Cache -> Simple OCR -> Vision API (Claude) -> Coordinate-based.
Each tier auto-logs hits to an Experience DB; repeat runs skip expensive tiers.
AppBot Eye -- Always-On Daemon
A single background process combining:
- Slack daemon (Socket Mode) -- live command delivery, thread-slot dashboard
- MCP broker -- exposes all CLI commands as JSON-RPC tools for Claude/Codex
- Hot-swap watchdog -- detects
wkappbot-core.new.exe, drains in-flight requests, renames atomically. Zero downtime on dotnet publish.
- Watchdog VBS -- if Eye itself dies for 2+ minutes, kills orphan cores and restarts.
Multi-AI Delegation
wkappbot ask triad "is this approach correct?" # GPT + Gemini + Claude in parallel
wkappbot ask claude "explain this chart" chart.png # vision-capable single ask
Triad runs thesis-antithesis-synthesis debate. Useful for architecture decisions, bug root causes, and code review -- anything where one model's blind spot is another's strength.
Skill System
Accumulated operator knowledge lives in versioned skills, queryable at any time:
wkappbot skill list
wkappbot skill read focusless-first-principle
wkappbot skill read grap
Skills capture per-project knowhow — UIA quirks, CDP gotchas, owner-drawn-control workarounds — so every session starts informed instead of exploring from scratch.
Suggest-Driven Backlog
wkappbot suggest "title: description" # queue a bug or improvement mid-task
wkappbot suggest list # review the backlog
wkappbot suggest resolve <ts> "note" --i-completed-... evidence.sh
AI agents queue findings without interrupting the current task. Evidence scripts are required to close a suggest -- no unverified resolves.
Architecture
┌─────────────────────────────────────────────────────────┐
│ Your terminal / Claude Code / Codex / any AI agent │
└───────────────────────┬─────────────────────────────────┘
│ wkappbot <command>
▼
┌─────────────────────────────────────────────────────────┐
│ wkappbot.exe (MIT launcher, ~1 MB, AOT) │
│ • routes CLI args → core via named pipe │
│ • hot-swap: detects .new.exe, drains, renames atomic │
│ • license check via GitHub collaborator API │
└───────────────────────┬─────────────────────────────────┘
│ named pipe wkappbot_eye_ipc_{hash}
▼
┌─────────────────────────────────────────────────────────┐
│ wkappbot-core.exe (closed, ~25 MB, single-file) │
│ • all CLI commands: a11y, ask, skill, eye, file, … │
│ • AppBot Eye daemon: Slack socket + MCP broker │
│ • UIA / Win32 / CDP / ADB automation engines │
│ • Vision / OCR pipeline + Experience DB │
└─────────────────────────────────────────────────────────┘
│ UIA / Win32 │ CDP (DevTools)
▼ ▼
Windows apps Chrome / Edge
(WPF, MFC, UWP) (web apps, AI chat)
Per-repo isolation: each git clone gets its own Eye instance and DataDir ({root}/.wkappbot/hq/).
📚 Documentation
→ kiexpert.github.io/wkappbot-sdk
60-second quickstart
git clone https://github.com/kiexpert/wkappbot-sdk %USERPROFILE%\Documents\wkappbot
cd %USERPROFILE%\Documents\wkappbot
build.cmd
Then add bin\ to PATH (see INSTALL.md), open a new terminal, and:
wkappbot --version # verify install
wkappbot find "*" # list open windows (title + UIA search)
wkappbot skill list # browse built-in knowhow
wkappbot a11y find "*Notepad*" # resolve a window element
Installation
Clone = install. That's it.
git clone https://github.com/kiexpert/wkappbot-sdk %USERPROFILE%\Documents\wkappbot
cd %USERPROFILE%\Documents\wkappbot
build.cmd
Pre-built binaries are also available if you prefer not to build:
Recommended layout — clone under Documents\ so your personal automation data (experience DB, logs, skills) stays in your home directory and is easy to find, back up, or exclude from sharing:
Why Documents? WKAppBot learns from your usage and stores UI experience data under bin\wkappbot.hq\. Keeping this under your personal Documents folder protects your privacy — it stays separate from shared or version-controlled paths.
%USERPROFILE%\Documents\wkappbot\ ← recommended clone location (easy to find in Explorer)
bin\
wkappbot.exe ← launcher / busybox entry point
wkappbot-core.exe ← core worker (hot-swapped on update)
wkappbot.hq\ ← runtime data: skills, logs, experience DB (auto-created)
csharp\
handlers\
skills\
...
Add bin\ to your PATH so wkappbot is available in any terminal:
# PowerShell (permanent, current user)
[Environment]::SetEnvironmentVariable(
'PATH',
"$env:USERPROFILE\Documents\wkappbot\bin;$([Environment]::GetEnvironmentVariable('PATH','User'))",
'User'
)
Build from source (requires .NET SDK 8+):
build.cmd
Verify the install:
wkappbot --version
wkappbot skill list
Requirements: Windows 10 22621+ (64-bit). No separate .NET runtime needed — the binary is self-contained.
Activate License
Free tier works out of the box — no signup. To unlock CDP browser automation,
multi-AI ask, schedule, or --sudo admin access:
gh auth login # authenticate with GitHub
wkappbot license status # confirms current tier (Free until you subscribe)
Then follow SUBSCRIBE.md — KIS bank transfer with your GitHub
username as the memo, accept the GitHub collaborator invite, and the same binary
unlocks Pro features within 1 hour. See PRICING.md for tier details.
CLI at a Glance
# Discovery
wkappbot a11y inspect <grap> # dump UIA tree
wkappbot a11y find <grap> # resolve + print # TARGET
wkappbot a11y highlight <grap> # visualize with zoom overlay
wkappbot find <keyword> # unified window-title + UIA element search
wkappbot scan <window-title> # scan app structure, build Experience DB
# Interaction (focusless where possible)
wkappbot a11y click <grap>
wkappbot a11y type <grap> "text"
wkappbot a11y read <grap> # CDP-first on browser windows, UIA fallback
wkappbot a11y scroll <grap> down 3
wkappbot dismiss "<window-title>" # auto-dismiss popups (OCR importance check)
# Browser / CDP
wkappbot a11y read "{proc:'chrome'}" # full page text via CDP
wkappbot a11y click ".submit-btn" # CSS selector
wkappbot a11y read "{proc:'chrome'}" --eval-js "document.title"
# Scenarios & high-level automation
wkappbot run scenario.yaml # run a YAML test scenario
wkappbot do "<window>" <form> <button> # full combo-select + click + dialog flow
# AI delegation
wkappbot ask triad "is this lock-free?"
wkappbot ask claude "explain this chart" chart.png
# Daemon & context
wkappbot eye tick # one-shot status + ctx%
wkappbot newchat "continue from: ..." # handoff to fresh session
wkappbot claude-usage # JSONL size + context %
YAML Scenarios
Describe multi-step test flows in plain YAML:
scenario: { name: "Order placement" }
app: { launch: "trading.exe", wait_for_window: { title_contains: "HTS" } }
steps:
- { name: "Enter stock code", target: { automation_id: "codeEdit" }, action: type_text, params: { text: "005930" } }
- { name: "Click buy", target: { name: "매수" }, action: click }
- { name: "Verify", target: { automation_id: "resultLabel" }, action: assert,
params: { type: text_contains, expected: "주문완료" } }
Architecture (module layout)
WKAppBot.CLI CLI entry, command routing, grap resolution
WKAppBot.Core ScenarioRunner, ActionExecutor, AAR readiness
WKAppBot.Win32 NativeMethods, WindowFinder, UiaLocator, SendInput tiers
WKAppBot.Vision ChartAnalyzer, SimpleOcrAnalyzer, VisionCache, CCA
WKAppBot.WebBot CdpClient, ChromeLauncher, SlackSocketClient
WKAppBot.Android AdbClient, AndroidA11yTree
WKAppBot.Launcher Hot-swap staging, pipe relay, admin elevation
Eye (always-on) ↔ MCP worker (Core) over JSON-RPC named pipe.
UIA isolated in a separate MCP worker process -- prevents ConPTY LPC deadlock.
Runtime
bin/wkappbot.exe official launcher (alias: a11y.exe)
bin/wkappbot-core.exe core worker (auto hot-swapped on publish)
bin/wkappbot.hq/ runtime HQ -- experience DB, skills, sessions, logs
.NET 8.0 · net8.0-windows10.0.22621.0 · Korean UI support
Trademark
"WKAppBot" and the AppBot Eye logo are trademarks of kivilab.co.kr.
You may use the name to accurately describe the software (e.g. "built with WKAppBot").
Do not use the name to imply endorsement or affiliation without written permission.
References
- CLAUDE.md -- detailed operational guidance for AI agents
- AGENTS.md -- shared AI engineering rules
wkappbot skill list — accumulated knowhow: grap syntax, UIA quirks, CDP gotchas, and more
Support This Project
If WKAppBot saves you time, consider buying me a coffee ☕
⭐ Star History & Community
- 💬 Questions or design discussions? Join us on GitHub Discussions.
- 🐛 Found a bug or have a feature request? Open an issue.
- 🛠 Built something with WKAppBot? Open a PR to add your project to a community showcase — we'd love to feature it.
- ⭐ Liking the project? Drop a star — it genuinely helps surface this work to other developers.
Encoding Policy
Repository text assets should be treated as UTF-8 by default.
- Prefer UTF-8 when creating or updating
.md, .txt, .html, .json, .yml, .xml, and other text-based files.
- If an imported source arrives in another encoding such as CP949, keep the original file for archival purposes and also store a UTF-8 converted copy for reading and editing.
- Use file-editing tools that preserve encoding correctly. If a tool may re-encode content incorrectly, prefer
wkedit or another UTF-8-safe path.
- Binary files such as
.pdf, .mp4, images, and archives should be preserved as-is and not text-converted.
- Multibyte filenames are allowed, but repository writes should still use UTF-8 so downstream tools and web viewers can read them reliably.
