English · 日本語

Bajutsu

Natural-language-driven E2E (end-to-end) testing built on a backend-agnostic driver: one scenario format and one deterministic runner, where a platform is a backend behind that one interface. Swap the backend and the same scenarios run on a different target — the iOS Simulator (XCUITest), a web (Playwright) backend, and an Android (adb) backend are all landed; Flutter apps run on those same iOS/Android backends unchanged, needing no new backend (BE-0008). Status: pre-alpha. The deterministic core, the AI authoring loop (record / crawl), the evidence subsystem, codegen, and self-healing triage are all implemented and unit-tested (no Simulator needed). The iOS XCUITest backend is validated end-to-end on a real Simulator — scenarios, evidence capture, and the triage self-heal loop all run on-device. The web (Playwright) backend runs a deterministic run against a browser on the Linux gate (demos/web), and the Android (adb) backend is validated end-to-end on an emulator (android-e2e.yml). A hosted server backend — a hostable control plane plus Mac and Linux workers, for teams that want serve shared instead of local — has also landed (docs/self-hosting.md).

Bajutsu takes test scenarios written in (or recorded from) natural language, drives your app — taps / text / swipes / waits — and verifies the result with machine-checkable assertions. Everything but one seam is platform-neutral: the scenario format, selector resolution, the deterministic runner, the evidence subsystem, and the reporter never name a platform. That one seam is the backend — the driver that actuates the UI. Point the runner at a different backend and the same scenario runs on a different target: the iOS Simulator (XCUITest), a browser (Playwright), or Android (adb). Flutter apps ride those same iOS/Android backends unchanged, needing no new backend. Choosing a platform is choosing a backend, not adopting a different tool.

The name. Bajutsu (馬術) is Japanese for horsemanship / equestrianism. The name refers to the sources of test instability the tool tames — flaky timing, async transitions, and unexpected system alerts on the iOS Simulator. Bajutsu drives the target deterministically through a scenario so that each run produces the same result — on the Simulator, and on every backend behind the same driver.

The central design decision is to keep the LLM (large language model) out of the CI (continuous integration) gate:

• AI is the author and the failure investigator, never the judge. It helps write scenarios (explore + record) and investigate failures, but a run is fully deterministic with no AI involved — pass/fail comes only from machine assertions.
• Two tiers. Tier 1 = AI live operation (exploration / authoring). Tier 2 = a deterministic runner for CI regression.

Design rationale (in Japanese) lives in DESIGN.md. Implementation-grounded, per-feature documentation lives in docs/ — English, with a Japanese mirror under docs/ja/.

Core principles

• Determinism first. Condition waits only (the one fixed pause is an explicit, capped sleep step that must state its reason); an ambiguous selector fails immediately instead of "tapping whatever matched first"; each test starts from a clean environment.
• Stable selectors. Prefer a non-localized, data-derived id — accessibilityIdentifier on iOS, data-testid on the web — over text; coordinates are the last resort.
• Stability ladder. Bajutsu attempts UI actions most-stable-first (semantic tap by id → coordinate tap → … ), and the chosen backend is the most stable one available.
• A platform is a backend. The deterministic core names no platform; the one platform-specific seam is the backend (xcuitest / playwright / adb / …) behind the Driver interface. Add or swap a backend and the same scenario format, runner, and CLI target a new platform unchanged — per-target and per-platform differences live only in config and the chosen backend.
• Evidence as rules. "Capture on every X" is normalized into reusable rules so the second run reproduces the same evidence without AI.

Architecture

Entry points share the scenario format: record and crawl (AI authoring / exploration), run (deterministic replay), and codegen (emit a native test). The deterministic core — selector resolution, the orchestrator, evidence, config, and reporting — lives under bajutsu/common/. The hosted control plane and the local web UI that can invoke it live under bajutsu/serve/. An import-linter contract keeps the two apart, so the core never depends on the hosting layer. The module list, the dependency layers, and the editable mermaid source are all in docs/architecture.md; the per-feature breakdown is in docs/.

Status

Implemented and covered by tests (run without a Simulator):

• Driver abstraction and selector resolution (the determinism core)
• Platform-aware backend registry — a target's required platform: decides its actuator: ios resolves to xcuitest (the sole iOS actuator since BE-0290 retired idb), and android / web / fake each to their own. --backend checks the target's platform, that platform's actuator, or fake
• Scenario schema: steps — tap / type / swipe / drag / scroll / double-tap / pinch / rotate / web for WebView content / device control / http / totp / email / generate / manual — waits, mid-step assert, reusable components (use), control flow (if / forEach), variables (extract → ${vars.*}), parametrization (data / dataFile), preconditions, permissions, interrupts, systemAlertHandling, network mocks, a network filter, capturePolicy evidence rules, and redact — with strict validation, YAML round-trip, and a generated JSON Schema
• Assertion evaluation (exists / value / label / count / enabled / disabled / selected / request / requestSequence / event / responseSchema / clipboard / golden / visual)
• Tier 2 run loop (act → wait → verify), tested via an in-memory fake driver
• Evidence subsystem: instant captures (screenshot / elements / actionLog), video / deviceLog / appTrace interval captures, network observation + mocks, visual regression (baselines + approve), a golden element-tree comparison, capturePolicy trigger rules, and secret redaction
• Reporting (manifest.json + JUnit XML + CTRF JSON + self-contained interactive HTML)
• Config resolution (team defaults × per-target; iOS or Android app.id, web app.url) and backend selection (the target's platform), with a per-platform JSON Schema from bajutsu config schema

Validated on a real Simulator (iPhone 17 Pro, recent iOS):

• The XCUITest backend — the resident runner's element snapshot of the XCTest automation tree, semantic tap / text / multi-touch / text selection driven by bundle id, and the simctl launch sequencing — confirmed by running the showcase scenarios, evidence capture, and the triage self-heal loop on-device via the XCUITest runner (built from the repo with Xcode's xcodebuild).

Validated in a browser (Linux, no Mac):

• The Playwright backend runs the demos/web scenarios deterministically inside the same gate as CI — proving the core is platform-neutral, including the richer web-only capabilities (network capture / video / multi-touch / parallel).

Validated on an Android emulator (Linux, no Mac):

• The adb backend's element reads (a resident UI Automator server, with uiautomator dump as the fallback), device-side re-resolved tap (falling back to a host-computed frame-center coordinate tap), and launch sequencing — actuation-fidelity parity with XCUITest — are confirmed against a booted emulator (API 34, under KVM) in android-e2e.yml, driving the same shared scenarios XCUITest runs.

Validated against a real Postgres (Linux, no Mac):

• The hosted server backend's Alembic migrations and its object-relational mapper (ORM) repository layer run against an ephemeral postgres:16 container in serve-db.yml — a required check, promoted from signal once it proved stable.

Not yet wired: the external services.mockServer command (superseded by in-scenario mocks); appTrace interval evidence on the web backend (iOS-only; the web backend has its own video / deviceLog-equivalent captures instead); nativeZ z-order reporting on SwiftUI/Compose screens (a declarative-toolkit limitation, not a Bajutsu gap). See docs/architecture.md for the full implemented-vs-unwired table.

Requirements

• Python 3.13 (managed via uv) — the deterministic core and the whole gate run anywhere, Linux included
• For iOS: macOS with Xcode (the iOS Simulator and xcodebuild) — the XCUITest runner builds from the repo
• For web: any OS with Playwright's Chromium (playwright install chromium) — no Mac needed
• For Android: any OS with adb and a booted device or emulator — no Mac needed (CI validates on API 34)
• For the hosted server backend: a Linux node for the control plane (Postgres, S3-compatible storage) plus Mac or Linux workers, or both — see docs/self-hosting.md. Optional: the local bajutsu serve needs none of it

Setup

New here? The Getting started tutorial walks the whole loop on an iOS Simulator. No Mac? The web track does the same loop against a browser (Playwright backend) on any OS — no Xcode or Simulator.

make setup                 # base: .venv (Python 3.13) + dev tools + git hooks (no backend, runs anywhere)
make install               # base PLUS exactly the backends your config needs (config-aware)

make setup is the config-agnostic floor the deterministic gate needs. make install builds on it: it reads your --config (pass one via make install ARGS="--config demos/showcase/showcase.config.yaml"), resolves which backends its targets.* actually use plus whether an AI provider is configured, and installs only those pip extras and external tools (the XCUITest runner built via Xcode's xcodebuild for iOS, Playwright's browser for web, the anthropic SDK when AI is configured) — idempotently, so it is safe to re-run. With no config in the working directory it installs nothing beyond the base. The requirements it draws from live in one mapping (bajutsu/common/provisioning/), shared with doctor's pre-flight so the two never drift.

Installing from PyPI instead? The base package is AI-free: pip install bajutsu gets the deterministic authoring / running paths with no AI SDK, and pip install bajutsu[ai] (or bajutsu[bedrock]) adds the SDK for the Claude paths — see What uses Claude. The base package also carries the server stack. With it, bajutsu serve runs on one machine with no extra. For a hosted deployment, the server extra adds boto3 and the db extra adds psycopg.

Usage

The CLI surface (full reference in docs/cli.md):

bajutsu run    --target <name> [--scenario file.yaml]        # default: the app's whole scenarios dir
bajutsu record --target <name> --goal "..." [--out file]     # AI explore + record (Tier 1, needs API key/login)
bajutsu crawl  --target <name> [--max-screens N]             # breadth-first crawl → screen map (Tier 1)
bajutsu doctor --target <name>                               # environment + convention score for the current screen
bajutsu codegen <scenario.yaml> --target <name> -o UITests/Foo.swift   # emit a native XCUITest
bajutsu approve --baselines <dir> [--scenario s.yaml]     # promote captured screenshots to visual baselines
bajutsu serve  [--port 8765] [--config c.yaml]            # local web UI: author + run + reports (Tier 1)
bajutsu mcp    [--config c.yaml] [--transport stdio]      # MCP server for agent integration (needs `bajutsu[mcp]`)
bajutsu lint   <scenario.yaml>                            # validate a scenario without running it
bajutsu schema                                            # print the JSON Schema for editor integration

trace (inspect a finished run), report (re-render report.html / JUnit / CTRF from stored data), triage (diagnose a failure), and worker (lease queued runs from the hosted server backend) round out this walkthrough. The CLI also wires audit / coverage / export / flakiness / impact / stats — see the CLI reference for the full command set.

make serve (or scripts/serve.sh) wraps bajutsu serve and installs the configured backend's dependencies on demand, so a fresh checkout will not hit no available actuator among ['xcuitest']. Pass flags via make serve ARGS="--port 8766". The server queues each run for a bajutsu worker, so make serve also starts one worker beside it and stops that worker with the server. SERVE_WORKER=0 make serve starts the server alone.

Per-app (and per-platform) settings live in a config file you pass with --config; the demos ship ready-to-run ones (e.g. demos/showcase/showcase.config.yaml, demos/web/demo.config.yaml). Each target names its platform, which picks the actuator. app.id names an iOS app, and app.url names a web app:

defaults:
  platforms:
    ios:
      runsOn: { ios: { locale: en_US } }

targets:
  showcase-swiftui:         # iOS app — driven on the Simulator via XCUITest
    platform: ios
    app:
      id: com.bajutsu.showcase.ios.swiftui
      launch: { env: { SHOWCASE_UITEST: "1" } }
      idNamespaces: [stable, horse, search, log, notice, perm, sys, net]
    paths: { scenarios: demos/showcase/scenarios }

  web:                      # web app — driven in a browser via Playwright
    platform: web
    app: { url: "http://127.0.0.1:8787/index.html" }
    paths: { scenarios: demos/web/scenarios }

BE-0452 explains the target keys and their grouping.

Demos

Runnable demos, all through one entry point — make -C demos <target> (demos/):

• tour — make -C demos tour. The whole lifecycle (run → modify → diagnose) on a real Simulator, fully deterministic, no API key. (It also runs against an in-memory fake device with zero setup: uv run python demos/tour/tour.py.)
• features — make -C demos features. The scenario-authoring features (tags, parameterized shared steps, secrets) on a real Simulator.
• webui — make -C demos webui. The Web UI driving a Simulator and collecting every evidence type: screenshots, video, logs, network (observed + mocked), visual regression, system-alert handling. The headline demo for iOS developers.
• record — make -C demos record. AI authoring with real Claude on a booted app, then the modify-and-self-heal (triage) loop.
• web — make -C demos/web e2e. The Playwright backend running scenarios against a static web app — no Mac or Simulator, runs on Linux. For a step-by-step walkthrough of this demo, see the web getting-started track.
• docs-site — bajutsu run --target docs --backend web --config demos/docs-site/docs-site.config.yaml. The Playwright backend driving the public, live docs site itself — no local app to serve, the target is a live URL.
• serve-ui — . The Playwright backend dogfooding the web UI's own single-page app, a deterministic regression net for the Web UI, no Mac or Simulator needed.

Development

make check                # the full gate: format + lint + typecheck + tests (mirrors CI exactly)
uv run pytest -q          # just the tests (no Simulator)

See CLAUDE.md and CONTRIBUTING.md for the working agreement.

Project layout

bajutsu/
├── common/               # deterministic core + shared periphery: drivers, scenario schema,
│                         #   orchestrator, evidence, config, backend command layer, AI seam,
│                         #   provisioning, analytics, github integration, cancellation, devices
│   ├── drivers/          #   Driver protocol + selector resolution; fake / xcuitest (iOS,
│   │                     #     incl. a live-device route) / playwright (web) / adb (Android)
│   ├── scenario/         #   scenario schema (models), YAML round-trip, expansion, JSON Schema,
│   │                     #     ${namespace.key} interpolation, system-alert config
│   ├── orchestrator/     #   deterministic Tier 2 run loop (act → wait → verify)
│   ├── runner/           #   config + scenarios -> report via a device pool
│   ├── evidence/         #   instant / interval capture, network observation, visual
│   │                     #     regression, golden element-tree comparison, secret redaction
│   ├── report/           #   manifest.json + JUnit + CTRF + interactive HTML
│   ├── config/           #   team defaults × per-target resolution
│   ├── ai/               #   vendor-neutral AiBackend seam (Anthropic API / Bedrock / CLI /
│   │                     #     Claude Code / disabled)
│   ├── agents/           #   authoring-agent periphery built on the ai/ seam: record / enrich
│   │                     #     / triage agents, system-alert guard
│   ├── backend_cli/      #   simctl (iOS) + adb (Android) command layers
│   ├── doctor/           #   convention score
│   ├── capability/       #   environment preflight for doctor / CI
│   └── github/           #   GitHub integration: Actions annotations, App install token
├── run/                  # `bajutsu run`'s CLI dispatch: target/backend resolution, device
│                         #   leasing, report dispatch, run-completion notifications (notify/)
├── record/               # AI record loop: observe -> propose -> execute -> emit a scenario
├── crawl/                # breadth-first crawl -> screen map (deterministic core/, AI-assisted
│                         #   guide/ and tabs/, report/ layout)
├── repl/                 # AI-free manual shell: read the element tree, act on an id
├── triage/               # self-healing triage: a rule-based heuristic agent (default) plus a
│                         #   Claude-backed agent (--ai)
├── codegen/              # scenario -> native tests (XCUITest / Playwright / UI Automator)
├── analysis/             # read-only advisory analysis, never gates CI: audit / coverage /
│                         #   flakiness / impact / stats / trace
├── serve/                # local web UI (author + run + reports; Tier 1) and, in server/, the
│                         #   hosted control plane (FastAPI + Postgres + OAuth); every job
│                         #   runs on a `bajutsu worker`
├── mcp/                  # MCP server (tools + resources for agent integration)
├── templates/            # Jinja report/dashboard templates + the web UI's own JS/CSS
├── cli/                  # CLI (typer) — one file per command
└── __main__.py

Roadmap

Bajutsu grows along three axes that compose independently: reach (more platforms and surfaces), scale & collaboration (from a local tool to a shared, hosted service), and authoring & maintenance (lowering the cost of owning tests). The deterministic runner, the AI record loop, capturePolicy evidence rules, codegen, and self-healing triage were the first to land — see Status above for the implemented surface. Since then, the web (Playwright) and Android (adb) backends and Flutter support have shipped on the reach axis. Both a public and a self-hosted control-plane topology have shipped on the scale axis. Every direction still holds to the same invariant: no future feature may put AI into the Tier 2 run / CI gate. The rationale behind each axis, and how far each has progressed, is in docs/vision.md.

The forward-looking, prioritized backlog (what we want to build next) lives in roadmaps/.

License

Licensed under the Apache License, Version 2.0. See NOTICE for attribution.