Run Espresso, XCUITest and Maestro tests on real devices in the cloud.
- Real Devices — Test on thousands of real iOS and Android devices
- Emulators & Simulators — Fast feedback with virtual devices
- Parallel Execution — Split tests across multiple devices with sharding
- CI/CD Ready — Integrates with GitHub Actions, Jenkins, and more
- Live Results — Watch tests run in real-time
- Artifacts — Download videos, screenshots, and logs
Installation
npm install -g @testingbot/cli
Requirements: NodeJS 20 or higher
Authentication
The CLI requires TestingBot API credentials. You can authenticate in several ways:
Browser Login (Recommended)
testingbot login
This opens your browser for authentication. After logging in, your credentials are saved to ~/.testingbot.
Other Methods
- Command-line options:
--api-key and --api-secret
- Environment variables:
TB_KEY and TB_SECRET
- Config file: Create
~/.testingbot with content key:secret
Command-line options win over the environment variables, which win over the file.
Check or remove credentials
testingbot whoami # account, plan, minutes left, concurrency, and which credentials are used
testingbot whoami --json
testingbot logout # deletes ~/.testingbot (TB_KEY / TB_SECRET are not touched)
List devices
testingbot devices lists the device names and OS versions each framework can run on, in the form --device, --deviceVersion / --platform-version and --device-matrix accept. No credentials are needed, except with --real.
testingbot devices # Maestro emulators and simulators
testingbot devices --framework xcuitest # or espresso
testingbot devices --platform ios --real # real devices, with live availability
testingbot devices --json
Commands
Maestro
Run Maestro UI tests on real devices and emulators/simulators.
testingbot maestro <app> <flows...> [options]
Arguments:
app - Path to your app file (.apk, .ipa, .app, .zip, or an EAS iOS simulator .tar.gz)flows - One or more paths to flow files (.yaml/.yml), directories, .zip files, or glob patterns
App Options:
| Option | Description |
|---|
--app <path> | Path to the application under test (alternative to the positional app argument) |
--other-app <path-or-url> | Additional companion app to install on the device alongside --app. Accepts a local file path (.apk, .ipa, .app, .zip) or a tb://<appkey> / http(s)://... URL — local paths are uploaded; URLs are passed through to the run as-is. Repeatable, max 4 entries. |
--app-url <url> | Download the app from an http(s) URL instead of a local file: .apk, .ipa, .zip or an EAS Build iOS .tar.gz (the .app inside is extracted automatically). Every positional argument is then a flow. Signed URLs such as EAS links expire after about an hour, so pass a fresh one |
--app-binary-id <projectId> | Reuse the app of a project uploaded earlier (testingbot upload, or any previous run's Project ID) instead of uploading one. Every positional argument is then a flow. The platform is taken from the stored app unless --platform is given |
Device Options:
| Option | Description |
|---|
--device <name> | Device name (e.g., "Pixel 9", "iPhone 16") |
--platform <name> | Platform: Android or iOS |
--deviceVersion <version> | OS version (e.g., "14", "17.2") |
--real-device | Use a real device instead of emulator/simulator |
--device-matrix <cells> | Run every flow on each listed device in one go. Cells are <device>[:<version>][:real], comma-separated or repeatable. Cannot be combined with --device or --deviceVersion; --real-device (or an .ipa app) applies to every cell |
--orientation <orientation> | Screen orientation: PORTRAIT or LANDSCAPE |
--disable-animations | Turn off device animations (Android animation scales, iOS Reduce Motion). --no-disable-animations keeps them on. Overrides platform.*.disableAnimations in config.yaml |
--device-locale <locale> | Device locale (e.g., "en_US", "de_DE") |
--timezone <timezone> | Timezone (e.g., "America/New_York", "Europe/London") |
Test Configuration:
| Option | Description |
|---|
--name <name> | Test name for dashboard identification |
--groups <names> | Tag the test session with one or more groups (comma-separated). Groups appear on the test in the TestingBot dashboard |
--include-tags <tags> | Only run flows with these tags (comma-separated) |
--exclude-tags <tags> | Exclude flows with these tags (comma-separated) |
--exclude-flows <paths> | Flow files, directories or glob patterns to leave out of the run (comma-separated, repeatable). An excluded flow that another flow still invokes via runFlow is bundled as a subflow but never runs on its own |
-e, --env <KEY=VALUE> | Environment variable for flows (can be repeated). The value is stored in plain text and shown in the run options, in the dashboard and in reports |
--secret-env <KEY=VALUE> | Like --env, but the value is stored encrypted and shown as *** in the run options, in the dashboard and in reports (can be repeated). A key may not be given to both flags |
--config <path> | Path to a custom Maestro config file (default: config.yaml in project root) |
--maestro-version <version> | Maestro version to use (e.g., "2.0.10") |
Values passed with -e are stored in plain text: they are returned in the run's options.env by the API, shown in the dashboard and embedded in the JUnit and HTML reports. Values passed with --secret-env are stored encrypted and never returned; the API, the dashboard and the reports show *** in their place. Use --secret-env for passwords, tokens and API keys. The CLI itself never prints --secret-env values, not even with --debug or --dry-run.
Network & Location:
| Option | Description |
|---|
--throttle-network <speed> | Network throttling: 4G, 3G, Edge, airplane, or disable |
--geo-country-code <code> | Geographic IP location (ISO country code, e.g., "US", "DE") |
Tunnel:
| Option | Description |
|---|
-t, --tunnel | Start a TestingBot tunnel for this test run (cannot be combined with --async) |
--tunnel-identifier <id> | Identifier for the tunnel, allowing multiple tunnels in parallel |
Output Options:
| Option | Description |
|---|
--async | Start tests and exit without waiting for results |
-q, --quiet | Suppress progress output |
--json | Print results as a single JSON document on stdout (logs move to stderr). Implies --quiet. Exit code 2 when tests fail |
--json-file | Write results as JSON to a file (default: <appId>_testingbot.json in the current directory). Implies --quiet. Exit code stays 0 when tests fail so the pipeline can gate on the file |
--json-file-name <path> | Custom path for the JSON results file (requires --json-file) |
--report <format> | Download report after completion: html, html-detailed, junit or allure |
--report-output-dir <path> | Directory to save reports (required with --report) |
--download-artifacts [mode] | Download test artifacts (logs, screenshots, video). Mode: all (default) or failed |
--artifacts-output-dir <path> | Directory to save artifacts zip (defaults to current directory) |
Advanced Options:
| Option | Description |
|---|
--shard-split <number> | Split flows into N parallel sessions for faster execution |
--retry <count> | Retry failed flows up to N times (0-2, default 0). Re-runs only the flows (or shards) that failed, the moment they fail, while the rest of the run continues. Cannot be combined with --async. |
--timeout <minutes> | Stop waiting for results after N minutes (default 60), counted from when the tests start and including --retry attempts. Unfinished runs are stopped and the command fails with exit code 2. Cannot be combined with --async. |
--fail-fast | Stop at the first failure that will not be retried: the runs still queued or running are cancelled, and the summary, reports and artifacts cover what ran. With --retry, a test only counts as failed once its retries are used up. Cannot be combined with --async. |
--ignore-checksum-check | Skip checksum verification and always upload the app |
Note on --retry: a failed flow/shard is retried immediately — as soon as it fails — without waiting for the other flows in the run to finish. Retry attempts appear live in the flow table marked with a ↻ icon. Each flow is retried independently up to N times, stopping as soon as that flow passes. Pass/fail uses the result of the last attempt per flow (last-attempt-wins), consistently across the CLI exit code, the TestingBot dashboard, and reports downloaded via --report.
CI/CD Integration:
| Option | Description |
|---|
--branch <name> | Git branch this test run was built from |
--commit-sha <sha> | Git commit SHA associated with this test run |
--pull-request-id <id> | Pull request ID this test run originated from |
--pr-url <url> | Pull request URL this test run originated from |
--repo-name <name> | Repository name (e.g., GitHub repo slug) |
--repo-owner <owner> | Repository owner (e.g., GitHub organization or username) |
-m, --metadata <KEY=VALUE> | Free-form metadata attached to the run and shown in the dashboard (repeatable, e.g. -m team=mobile -m env=staging) |
--check-name <name> | Name the GitHub pull request check TestingBot / <name> so several runs on one commit (iOS, Android) post separate checks that branch protection can require individually |
--cancel-previous | Before starting, cancel your queued or running runs of the same repository and branch (or pull request) from an earlier commit. Runs on the same commit are left alone |
Allure reports: --report allure converts each run's results into Allure result files under <report-output-dir>/allure-results/, one JSON per flow with its steps, status and failure details. Render them with allure serve <report-output-dir>/allure-results (requires the Allure CLI). Results from several runs or shards accumulate in the same directory.
Migrating from Maestro Cloud: the maestro cloud spelling of common flags is accepted as hidden aliases, so an existing command line runs unchanged: --app-file, --flows <a,b>, --apiKey, --device-model iPhone-17-Pro, --device-os iOS-18-2 / android-34, --format JUNIT|HTML, --output <file> (its directory becomes --report-output-dir) and --test-suite-name. The canonical flag wins when both are given.
Examples:
# Basic usage
testingbot maestro app.apk ./flows
# Multiple flow directories
testingbot maestro app.apk ./flows/smoke ./flows/regression ./flows/e2e
# With device selection
testingbot maestro app.apk ./flows --device "Pixel 8" --deviceVersion "14"
# Android app on real device with tags
testingbot maestro app.apk ./flows --device "Samsung Galaxy S24" --real-device --include-tags "smoke,regression"
# Tag the test session with groups (visible in the dashboard)
testingbot maestro app.apk ./flows --groups "smoke,critical"
# With environment variables: -e for plain values, --secret-env for credentials
testingbot maestro app.apk ./flows -e API_URL=https://staging.example.com --secret-env API_KEY=s3cretValue123
# With companion apps installed alongside the main app (up to 4)
# Each --other-app can be a local file (uploaded) or a tb:// / http(s):// URL (passed through)
testingbot maestro --app main.apk \
--other-app helper.apk \
--other-app tb://existing-appkey \
--other-app https://example.com/mock-server.apk \
./flows
# Download JUnit report
testingbot maestro app.apk ./flows --report junit --report-output-dir ./reports
# Download all artifacts (logs, screenshots, video)
testingbot maestro app.apk ./flows --download-artifacts --name "build-123"
# Download artifacts only for failed tests
testingbot maestro app.apk ./flows --download-artifacts failed --artifacts-output-dir ./artifacts
# Use a custom config file
testingbot maestro app.apk ./flows --config .maestro/ci-config.yaml
# Run in background (async)
testingbot maestro app.apk ./flows --async
# Split flows across 3 shards, grouping all flows over 3 parallel sessions
testingbot maestro app.apk ./flows --shard-split 3
# Retry failed flows up to 2 times (re-runs only the flows that failed)
testingbot maestro app.apk ./flows --retry 2
# CI/CD integration with Git metadata
testingbot maestro app.apk ./flows \
--commit-sha "abc123def" \
--pull-request-id "42" \
--repo-owner "myorg" \
--repo-name "myapp"
Device matrix
Run the same flows across several devices in a single command. Each cell names exactly one device; there is no cross-product, because not every device exists in every OS version. Every flow runs once per device, so the cost is devices × flows, and the CLI prints that summary before submitting.
testingbot maestro app.apk ./flows \
--device-matrix "Pixel 9:14" \
--device-matrix "Samsung Galaxy S24:14:real" \
--device-matrix "Pixel 8"
Each device becomes its own run with its own results, live table rows and dashboard link; --json lists the device per run, and --retry re-runs only the flow that failed on the device it failed on. If any cell is not a valid device/OS combination the whole request is rejected and nothing runs, so a matrix never partially submits.
Organizing flows and subflows
Every top-level flow you pass runs as its own test. A subflow (a reusable
flow another flow pulls in with runFlow) should not be passed as a
top-level flow — if it is, it runs twice: once standalone and once as part of
the flow that calls it.
Maestro has no notion of a "subflow-only" file. A .yaml sitting alongside
your real flows is a runnable flow, regardless of its name. Naming it
*.shared.yaml does not make Maestro treat it as shared.
Recommended structure — keep subflows in their own directory:
flows/
login.yaml # top-level, runs
checkout.yaml # top-level, runs
subflows/
sign-in.yaml # only runs when a flow calls it via runFlow
# flows/login.yaml
- runFlow:
file: subflows/sign-in.yaml
env:
APP_ID: com.example.app
Then pass only the directory of top-level flows:
# Runs login.yaml and checkout.yaml; sign-in.yaml is bundled automatically
# (as a runFlow dependency) but never runs on its own.
testingbot maestro app.apk ./flows
When you pass individual files, list only the flows you want to run — their
runFlow targets are discovered and uploaded for you:
# Correct: only the top-level flow. sign-in.yaml is bundled automatically.
testingbot maestro app.apk ./flows/login.yaml
# Wrong: sign-in.yaml would run twice.
testingbot maestro app.apk ./flows/login.yaml ./flows/subflows/sign-in.yaml
Other ways to keep a subflow out of a run:
config.yaml globs — list only the folders that hold top-level flows
(e.g. flows: ["*.yaml"]), leaving subflow folders out of discovery.- Tags — add
tags: [subflow] to the subflow's header and pass
--exclude-tags subflow.
Preview before you run. --dry-run prints exactly which flows run
standalone and which are bundled as runFlow subflows, without spending any
device minutes:
testingbot maestro app.apk ./flows --dry-run
testingbotctl also prints a warning if a flow you passed will run more than
once because another top-level flow calls it via runFlow.
Files the flows use
Files referenced by runFlow, runScript, addMedia and assertScreenshot are bundled automatically, relative to the flow that references them. For assertScreenshot, .png is added when the path has no extension, as Maestro does.
A path that contains a variable, such as screenshots/${DEVICE}/home.png, can't be resolved before the run. List the files under includedPaths in config.yaml. The globs are relative to the config file and must stay inside the flows directory:
# config.yaml
flows:
- 'flows/*'
includedPaths:
- 'screenshots/**'
--dry-run lists every bundled file and warns about references that won't be in the upload.
Upload once, run many times
testingbot upload pushes an app once and prints a Project ID. Later runs pass that ID with --app-binary-id and skip the upload entirely; each run still gets its own project and results.
testingbot upload app.apk
# Uploaded app.apk. Project ID: 4321
# Run flows against it with: testingbot maestro --app-binary-id 4321 ./flows
APP_ID=$(testingbot upload app.apk --json | jq -r .appId)
testingbot maestro --app-binary-id "$APP_ID" ./flows/smoke
testingbot maestro --app-binary-id "$APP_ID" ./flows/regression --device "Pixel 9"
Every maestro run also prints its Project ID after the app upload, so any previous run's ID works with --app-binary-id too. Unchanged binaries are deduplicated by checksum on upload as well; pass --ignore-checksum-check to force a fresh upload.
upload <appFile>
| Option | Description |
|---|
--ignore-checksum-check | Skip checksum verification and always upload the app |
-q, --quiet | Suppress upload progress |
--json returns { provider, appId, file, url }. Fails with exit code 1 if the upload was rejected.
Expo / EAS Build
eas build produces a download URL rather than a local file, and for iOS simulator builds the artifact is a .tar.gz containing the .app. Both are handled directly:
# iOS simulator build from EAS (tar.gz with the .app inside)
URL=$(eas build --platform ios --profile preview --json --non-interactive | jq -r '.[0].artifacts.buildUrl')
testingbot maestro --app-url "$URL" ./flows --device "iPhone 16"
# Android build from EAS
URL=$(eas build --platform android --profile preview --json --non-interactive | jq -r '.[0].artifacts.buildUrl')
testingbot maestro --app-url "$URL" ./flows --device "Pixel 9"
# A local tar.gz works too
testingbot maestro build.tar.gz ./flows
Inside an EAS Build or EAS Workflows job, the run is tagged automatically with the EAS build id, profile and platform, and EAS_BUILD_GIT_COMMIT_HASH is used as the commit SHA unless --commit-sha is given. testingbot upload <url> accepts URLs as well.
Status, artifacts and list
Commands for working with Maestro projects after they were started, typically together with --async. Every command accepts --api-key / --api-secret, --debug, and the --json, --json-file, --json-file-name output flags described under JSON Output.
# Start tests without waiting and capture the project id
testingbot maestro app.apk ./flows --async --json | jq -r .appId
# Check on it later; --wait blocks with live progress and exits 2 on failure
testingbot status --id 1234
testingbot status --id 1234 --wait
# Fetch reports and artifacts once it finished
testingbot artifacts --id 1234 --report junit --report-output-dir ./reports
testingbot artifacts --id 1234 --download-artifacts failed --artifacts-output-dir ./artifacts
# Browse recent projects
testingbot list
testingbot list --count 25 --offset 25 --json
status --id <projectId>
| Option | Description |
|---|
-w, --wait | Block until every run has finished, showing the same live flow table as a foreground run. Ctrl-C detaches without cancelling the runs |
-q, --quiet | Suppress progress output |
Exit code is 0 while the project is still running (JSON outcome: "running"), 0/2 once it completed, 1 on errors.
artifacts --id <projectId>
| Option | Description |
|---|
--report <format> | Download report: html, html-detailed, junit or allure |
--report-output-dir <path> | Directory to save reports (required with --report) |
--download-artifacts [mode] | Download logs, screenshots and video. Mode: all (default) or failed |
--artifacts-output-dir <path> | Directory to save the artifacts zip (defaults to current directory) |
Fails with exit code 1 if the project is still running; use status --wait first.
list
| Option | Description |
|---|
--count <number> | Maximum number of projects to return (default 10) |
--offset <number> | Number of projects to skip, for pagination |
Projects are listed newest first with id, name, state, run and flow counts. --json returns { provider, meta: { offset, count, total }, projects: [...] } with a dashboard url per project.
Espresso
Run Android Espresso tests on real devices and emulators.
testingbot espresso [appFile] [testAppFile] [options]
Arguments:
appFile - Path to application APK filetestAppFile - Path to test APK file containing Espresso tests
Device Options:
| Option | Description |
|---|
--app <path> | Path to application APK file |
--test-app <path> | Path to test APK file |
--device <name> | Device name (e.g., "Pixel 6", "Samsung.*") |
--platform-version <version> | Android OS version (e.g., "12", "13", "14") |
--real-device | Use a real device instead of an emulator |
--tablet-only | Only allocate tablet devices |
--phone-only | Only allocate phone devices |
--locale <locale> | Device locale (e.g., "en_US", "de_DE") |
--timezone <timezone> | Timezone (e.g., "America/New_York", "Europe/London") |
Test Configuration:
| Option | Description |
|---|
--name <name> | Test name for dashboard identification |
--build <build> | Build identifier for grouping test runs |
--test-runner <runner> | Custom test instrumentation runner |
--language <lang> | App language (ISO 639-1 code, e.g., "en", "fr", "de") |
Test Filtering:
| Option | Description |
|---|
--class <classes> | Run tests in specific classes (comma-separated fully qualified names) |
--not-class <classes> | Exclude tests in specific classes |
--package <packages> | Run tests in specific packages (comma-separated) |
--not-package <packages> | Exclude tests in specific packages |
--annotation <annotations> | Run tests with specific annotations (comma-separated) |
--not-annotation <annotations> | Exclude tests with specific annotations |
--size <sizes> | Run tests by size: small, medium, large (comma-separated) |
Network & Location:
| Option | Description |
|---|
--throttle-network <speed> | Network throttling: 4G, 3G, Edge, or airplane |
--geo-location <code> | Geographic IP location (ISO country code, e.g., "US", "DE") |
Tunnel:
| Option | Description |
|---|
-t, --tunnel | Start a TestingBot tunnel for this test run (cannot be combined with --async) |
--tunnel-identifier <id> | Identifier for the tunnel, allowing multiple tunnels in parallel |
Output Options:
| Option | Description |
|---|
--async | Start tests and exit without waiting for results |
-q, --quiet | Suppress progress output |
--json | Print results as a single JSON document on stdout (logs move to stderr). Implies --quiet. Exit code 2 when tests fail |
--json-file | Write results as JSON to a file (default: <appId>_testingbot.json in the current directory). Implies --quiet. Exit code stays 0 when tests fail so the pipeline can gate on the file |
--json-file-name <path> | Custom path for the JSON results file (requires --json-file) |
--report <format> | Download report after completion: junit (Espresso has no HTML report) |
--report-output-dir <path> | Directory to save reports (required with --report) |
--retry <count> | Run a failed test again up to N times (0-2). The last attempt decides the result, the report and the artifacts |
--timeout <minutes> | Stop waiting for results after N minutes (default 60), counted from when the tests start and including --retry attempts. Unfinished runs are stopped and the command fails with exit code 2. Cannot be combined with --async. |
--fail-fast | Stop at the first failure that will not be retried: the runs still queued or running are cancelled, and the summary, reports and artifacts cover what ran. With --retry, a test only counts as failed once its retries are used up. Cannot be combined with --async. |
--download-artifacts [mode] | Download video, logs and screenshots as a zip after completion: all (default) or failed |
--artifacts-output-dir <path> | Directory for the artifacts zip (defaults to the current directory) |
Examples:
# Basic usage with positional arguments
testingbot espresso app.apk app-test.apk --device "Pixel 8"
# Using named options
testingbot espresso --app app.apk --test-app app-test.apk --device "Pixel 8"
# Real device with specific Android version
testingbot espresso app.apk app-test.apk \
--device "Samsung Galaxy S24" \
--platform-version "14" \
--real-device
# Run specific test classes
testingbot espresso app.apk app-test.apk \
--device "Pixel 8" \
--class "com.example.LoginTest,com.example.HomeTest"
# Run tests with annotations
testingbot espresso app.apk app-test.apk \
--device "Pixel 8" \
--annotation "com.example.SmokeTest" \
--size "small,medium"
# With network throttling and geolocation
testingbot espresso app.apk app-test.apk \
--device "Pixel 8" \
--throttle-network "3G" \
--geo-location "DE" \
--language "de"
# Download JUnit report
testingbot espresso app.apk app-test.apk \
--device "Pixel 8" \
--report junit \
--report-output-dir ./reports
XCUITest
Run iOS XCUITest tests on real devices and simulators.
testingbot xcuitest [appFile] [testAppFile] [options]
Arguments:
appFile - Path to application IPA filetestAppFile - Path to test ZIP file containing XCUITests
Device Options:
| Option | Description |
|---|
--app <path> | Path to application IPA file |
--test-app <path> | Path to test ZIP file |
--device <name> | Device name (e.g., "iPhone 15", "iPad.*") |
--platform-version <version> | iOS version (e.g., "17.0", "18.2") |
--real-device | Use a real device instead of a simulator |
--tablet-only | Only allocate tablet devices |
--phone-only | Only allocate phone devices |
--orientation <orientation> | Screen orientation: PORTRAIT or LANDSCAPE |
--locale <locale> | Device locale (e.g., "DE", "US") |
--timezone <timezone> | Timezone (e.g., "America/New_York", "Europe/London") |
Test Configuration:
| Option | Description |
|---|
--name <name> | Test name for dashboard identification |
--build <build> | Build identifier for grouping test runs |
--language <lang> | App language (ISO 639-1 code, e.g., "en", "fr", "de") |
Network & Location:
| Option | Description |
|---|
--throttle-network <speed> | Network throttling: 4G, 3G, Edge, or airplane |
--geo-location <code> | Geographic IP location (ISO country code, e.g., "US", "DE") |
Tunnel:
| Option | Description |
|---|
-t, --tunnel | Start a TestingBot tunnel for this test run (cannot be combined with --async) |
--tunnel-identifier <id> | Identifier for the tunnel, allowing multiple tunnels in parallel |
Output Options:
| Option | Description |
|---|
--async | Start tests and exit without waiting for results |
-q, --quiet | Suppress progress output |
--json | Print results as a single JSON document on stdout (logs move to stderr). Implies --quiet. Exit code 2 when tests fail |
--json-file | Write results as JSON to a file (default: <appId>_testingbot.json in the current directory). Implies --quiet. Exit code stays 0 when tests fail so the pipeline can gate on the file |
--json-file-name <path> | Custom path for the JSON results file (requires --json-file) |
--report <format> | Download report after completion: html or junit |
--report-output-dir <path> | Directory to save reports (required with --report) |
--retry <count> | Run a failed test again up to N times (0-2). The last attempt decides the result, the report and the artifacts |
--timeout <minutes> | Stop waiting for results after N minutes (default 60), counted from when the tests start and including --retry attempts. Unfinished runs are stopped and the command fails with exit code 2. Cannot be combined with --async. |
--fail-fast | Stop at the first failure that will not be retried: the runs still queued or running are cancelled, and the summary, reports and artifacts cover what ran. With --retry, a test only counts as failed once its retries are used up. Cannot be combined with --async. |
--download-artifacts [mode] | Download video, logs and screenshots as a zip after completion: all (default) or failed |
--artifacts-output-dir <path> | Directory for the artifacts zip (defaults to the current directory) |
Examples:
# Basic usage with positional arguments
testingbot xcuitest app.ipa app-test.zip --device "iPhone 16"
# Using named options
testingbot xcuitest --app app.ipa --test-app app-test.zip --device "iPhone 16"
# Real device with specific iOS version
testingbot xcuitest app.ipa app-test.zip \
--device "iPhone 15 Pro" \
--platform-version "17.2" \
--real-device
# iPad in landscape mode
testingbot xcuitest app.ipa app-test.zip \
--device "iPad Pro" \
--tablet-only \
--orientation LANDSCAPE
# With localization settings
testingbot xcuitest app.ipa app-test.zip \
--device "iPhone 16" \
--locale "DE" \
--language "de" \
--timezone "Europe/Berlin"
# With network throttling and geolocation
testingbot xcuitest app.ipa app-test.zip \
--device "iPhone 16" \
--throttle-network "3G" \
--geo-location "DE"
# Download HTML report
testingbot xcuitest app.ipa app-test.zip \
--device "iPhone 16" \
--report html \
--report-output-dir ./reports
# Run in background
testingbot xcuitest app.ipa app-test.zip \
--device "iPhone 16" \
--async
Common Features
CI/CD Metadata
maestro, espresso and xcuitest all take --branch, --commit-sha, --pull-request-id, --pr-url, --repo-owner, --repo-name and -m, --metadata <KEY=VALUE> (repeatable). Inside GitHub Actions, GitLab CI, Bitbucket Pipelines, Azure Pipelines or CircleCI, the repository URL, branch, commit and pull request are read from the CI environment, so the dashboard links the run to them on whichever host the repository lives (GitHub, GitLab, Bitbucket, Azure Repos or self-hosted). Any flag overrides what was detected.
testingbot maestro --cancel-previous uses this context to cancel the runs of a previous push: your queued or running Maestro runs with the same repository and the same pull request (or branch) but a different commit. Runs on the same commit, such as the iOS and Android jobs of one pipeline, keep running.
CI Job Summary and Annotations
In GitHub Actions the results are added to the job summary ($GITHUB_STEP_SUMMARY) as a table per device with links to TestingBot, and each failed flow or run gets an ::error annotation, shown on the workflow run and the pull request. In Azure Pipelines the same table is attached to the build (Extensions tab) and failures are logged as errors. Annotations are left out with --json, since stdout then holds the JSON document. Set TESTINGBOT_CI_SUMMARY=0 to turn this off.
Proxy
Behind a corporate proxy, the CLI follows the standard environment variables: HTTPS_PROXY (and HTTP_PROXY for http URLs, or ALL_PROXY), with NO_PROXY for hosts to reach directly. Or pass --proxy to any command:
testingbot maestro app.apk ./flows --device "Pixel 8" --proxy http://user:[email protected]:3128
API calls, uploads, live updates and the tunnel download all go through the proxy, and --tunnel hands it to the tunnel (--proxy / --proxy-userpwd). HTTP and HTTPS proxies are supported; HTTPS traffic is tunnelled with CONNECT. URL-encode special characters in the password (@ as %40).
Real-time Progress
By default, the CLI shows real-time progress updates including:
- Test status updates with actual device names (even when using wildcards)
- Device allocation status
- Live output from Maestro flows
Use --quiet to suppress progress output.
Graceful Shutdown
Press Ctrl+C to gracefully stop running tests. The CLI will:
- Stop all active test runs on TestingBot
- Clean up resources
- Exit with appropriate status code
Press Ctrl+C twice to force exit immediately.
Report Downloads
All test frameworks support downloading reports after completion. Maestro and XCUITest offer JUnit and HTML; Espresso offers JUnit only:
# JUnit XML format (for CI integration)
--report junit --report-output-dir ./reports
# HTML format (for human viewing)
--report html --report-output-dir ./reports
Artifact Downloads
Download all test artifacts including logs, screenshots, and video recordings (maestro, espresso and xcuitest):
testingbot maestro app.apk ./flows --download-artifacts --name "my-build"
testingbot xcuitest app.ipa tests.zip --download-artifacts failed
Artifacts are saved as a zip file named after the --name value (or with a timestamp if not provided), with a folder per run. --download-artifacts failed only fetches failed runs.
Exit Codes
| Code | Meaning |
|---|
0 | All tests passed (also for --async, --dry-run, and failed tests with --json-file) |
1 | CLI or infrastructure error: invalid arguments, missing credentials, upload failure, timeout |
2 | One or more tests failed |
Distinguishing 1 from 2 lets CI decide whether to retry the job or fail the build.
JSON Output
--json prints one JSON document on stdout and moves all log lines to stderr, so testingbot maestro app.apk ./flows --json | jq works. --json-file writes the same document to disk while keeping the normal console output. Both flags imply --quiet.
{
"provider": "maestro",
"outcome": "failed",
"success": false,
"appId": 1234,
"url": "https://testingbot.com/members/maestro/1234",
"runs": [
{
"id": 5678,
"status": "DONE",
"passed": false,
"device": { "name": "Pixel 6", "platform": "Android", "version": "14" },
"url": "https://testingbot.com/members/maestro/1234/runs/5678",
"flows": [
{
"id": 1,
"runId": 5678,
"name": "login",
"status": "DONE",
"passed": true,
"attempt": 1,
"latest": true,
"startedAt": "2026-01-01T00:00:00Z",
"completedAt": "2026-01-01T00:00:30Z",
"durationSeconds": 30,
"errors": []
}
]
}
]
}
outcome is one of passed, failed, started (--async), dry-run, or error. On error the document carries an error message and the exit code is 1.flows (Maestro only) lists every attempt, including --retry re-runs. attempt counts from 1; latest marks the attempt whose verdict counts for the run.runs is empty for --async, --dry-run, and errors raised before tests were submitted.
Documentation
For more information, visit TestingBot Documentation.
License
MIT
