Run Espresso, XCUITest and Maestro tests on real devices in the cloud.

• Real Devices — Test on thousands of real iOS and Android devices
• Emulators & Simulators — Fast feedback with virtual devices
• Parallel Execution — Split tests across multiple devices with sharding
• CI/CD Ready — Integrates with GitHub Actions, Jenkins, and more
• Live Results — Watch tests run in real-time
• Artifacts — Download videos, screenshots, and logs

Installation

npm install -g @testingbot/cli

Requirements: NodeJS 20 or higher

Authentication

The CLI requires TestingBot API credentials. You can authenticate in several ways:

Browser Login (Recommended)

testingbot login

This opens your browser for authentication. After logging in, your credentials are saved to ~/.testingbot.

Other Methods

• Command-line options: --api-key and --api-secret
• Environment variables: TB_KEY and TB_SECRET
• Config file: Create ~/.testingbot with content key:secret

Commands

Maestro

Run Maestro UI tests on real devices and emulators/simulators.

testingbot maestro <app> <flows...> [options]

Arguments:

• app - Path to your app file (.apk, .ipa, .app, or .zip)
• flows - One or more paths to flow files (.yaml/.yml), directories, .zip files, or glob patterns

App Options:

| Option | Description | |--------|-------------| | --app <path> | Path to the application under test (alternative to the positional app argument) | | --other-app <path-or-url> | Additional companion app to install on the device alongside --app. Accepts a local file path (.apk, .ipa, .app, .zip) or a tb://<appkey> / http(s)://... URL — local paths are uploaded; URLs are passed through to the run as-is. Repeatable, max 4 entries. |

Device Options:

| Option | Description | |--------|-------------| | --device <name> | Device name (e.g., "Pixel 9", "iPhone 16") | | --platform <name> | Platform: Android or iOS | | --deviceVersion <version> | OS version (e.g., "14", "17.2") | | --real-device | Use a real device instead of emulator/simulator | | --orientation <orientation> | Screen orientation: PORTRAIT or LANDSCAPE | | --device-locale <locale> | Device locale (e.g., "en_US", "de_DE") | | --timezone <timezone> | Timezone (e.g., "America/New_York", "Europe/London") |

Test Configuration:

| Option | Description | |--------|-------------| | --name <name> | Test name for dashboard identification | | --build <build> | Build identifier for grouping test runs | | --groups <names> | Tag the test session with one or more groups (comma-separated). Groups appear on the test in the TestingBot dashboard | | --include-tags <tags> | Only run flows with these tags (comma-separated) | | --exclude-tags <tags> | Exclude flows with these tags (comma-separated) | | -e, --env <KEY=VALUE> | Environment variable for flows (can be repeated) | | --config <path> | Path to a custom Maestro config file (default: config.yaml in project root) | | --maestro-version <version> | Maestro version to use (e.g., "2.0.10") |

Network & Location:

| Option | Description | |--------|-------------| | --throttle-network <speed> | Network throttling: 4G, 3G, Edge, airplane, or disable | | --geo-country-code <code> | Geographic IP location (ISO country code, e.g., "US", "DE") |

Tunnel:

| Option | Description | |--------|-------------| | -t, --tunnel | Start a TestingBot tunnel for this test run (cannot be combined with --async) | | --tunnel-identifier <id> | Identifier for the tunnel, allowing multiple tunnels in parallel |

Output Options:

| Option | Description | |--------|-------------| | --async | Start tests and exit without waiting for results | | -q, --quiet | Suppress progress output | | --report <format> | Download report after completion: html or junit | | --report-output-dir <path> | Directory to save reports (required with --report) | | --download-artifacts [mode] | Download test artifacts (logs, screenshots, video). Mode: all (default) or failed | | --artifacts-output-dir <path> | Directory to save artifacts zip (defaults to current directory) |

Advanced Options:

| Option | Description | |--------|-------------| | --shard-split <number> | Split flows into N parallel sessions for faster execution | | --retry <count> | Retry failed flows up to N times (0-2, default 0). Re-runs only the flows (or shards) that failed, the moment they fail, while the rest of the run continues. Cannot be combined with --async. | | --ignore-checksum-check | Skip checksum verification and always upload the app |

Note on --retry: a failed flow/shard is retried immediately — as soon as it fails — without waiting for the other flows in the run to finish. Retry attempts appear live in the flow table marked with a ↻ icon. Each flow is retried independently up to N times, stopping as soon as that flow passes. Pass/fail uses the result of the last attempt per flow (last-attempt-wins), consistently across the CLI exit code, the TestingBot dashboard, and reports downloaded via --report.

CI/CD Integration:

| Option | Description | |--------|-------------| | --commit-sha <sha> | Git commit SHA associated with this test run | | --pull-request-id <id> | Pull request ID this test run originated from | | --repo-name <name> | Repository name (e.g., GitHub repo slug) | | --repo-owner <owner> | Repository owner (e.g., GitHub organization or username) |

Examples:

# Basic usage
testingbot maestro app.apk ./flows

# Multiple flow directories
testingbot maestro app.apk ./flows/smoke ./flows/regression ./flows/e2e

# With device selection
testingbot maestro app.apk ./flows --device "Pixel 8" --deviceVersion "14"

# Android app on real device with tags
testingbot maestro app.apk ./flows --device "Samsung Galaxy S24" --real-device --include-tags "smoke,regression"

# Tag the test session with groups (visible in the dashboard)
testingbot maestro app.apk ./flows --groups "smoke,critical"

# With environment variables
testingbot maestro app.apk ./flows -e API_URL=https://staging.example.com -e API_KEY=secret

# With companion apps installed alongside the main app (up to 4)
# Each --other-app can be a local file (uploaded) or a tb:// / http(s):// URL (passed through)
testingbot maestro --app main.apk \
  --other-app helper.apk \
  --other-app tb://existing-appkey \
  --other-app https://example.com/mock-server.apk \
  ./flows

# Download JUnit report
testingbot maestro app.apk ./flows --report junit --report-output-dir ./reports

# Download all artifacts (logs, screenshots, video)
testingbot maestro app.apk ./flows --download-artifacts --build "build-123"

# Download artifacts only for failed tests
testingbot maestro app.apk ./flows --download-artifacts failed --artifacts-output-dir ./artifacts

# Use a custom config file
testingbot maestro app.apk ./flows --config .maestro/ci-config.yaml

# Run in background (async)
testingbot maestro app.apk ./flows --async

# Split flows across 3 shards, grouping all flows over 3 parallel sessions
testingbot maestro app.apk ./flows --shard-split 3

# Retry failed flows up to 2 times (re-runs only the flows that failed)
testingbot maestro app.apk ./flows --retry 2

# CI/CD integration with Git metadata
testingbot maestro app.apk ./flows \
  --commit-sha "abc123def" \
  --pull-request-id "42" \
  --repo-owner "myorg" \
  --repo-name "myapp"

Organizing flows and subflows

Every top-level flow you pass runs as its own test. A subflow (a reusable flow another flow pulls in with runFlow) should not be passed as a top-level flow — if it is, it runs twice: once standalone and once as part of the flow that calls it.

Maestro has no notion of a "subflow-only" file. A .yaml sitting alongside your real flows is a runnable flow, regardless of its name. Naming it *.shared.yaml does not make Maestro treat it as shared.

Recommended structure — keep subflows in their own directory:

flows/
  login.yaml            # top-level, runs
  checkout.yaml         # top-level, runs
  subflows/
    sign-in.yaml        # only runs when a flow calls it via runFlow

# flows/login.yaml
- runFlow:
    file: subflows/sign-in.yaml
    env:
      APP_ID: com.example.app

Then pass only the directory of top-level flows:

# Runs login.yaml and checkout.yaml; sign-in.yaml is bundled automatically
# (as a runFlow dependency) but never runs on its own.
testingbot maestro app.apk ./flows

When you pass individual files, list only the flows you want to run — their runFlow targets are discovered and uploaded for you:

# Correct: only the top-level flow. sign-in.yaml is bundled automatically.
testingbot maestro app.apk ./flows/login.yaml

# Wrong: sign-in.yaml would run twice.
testingbot maestro app.apk ./flows/login.yaml ./flows/subflows/sign-in.yaml

Other ways to keep a subflow out of a run:

• config.yaml globs — list only the folders that hold top-level flows (e.g. flows: ["*.yaml"]), leaving subflow folders out of discovery.
• Tags — add tags: [subflow] to the subflow's header and pass --exclude-tags subflow.

Preview before you run. --dry-run prints exactly which flows run standalone and which are bundled as runFlow subflows, without spending any device minutes:

testingbot maestro app.apk ./flows --dry-run

testingbotctl also prints a warning if a flow you passed will run more than once because another top-level flow calls it via runFlow.

Espresso

Run Android Espresso tests on real devices and emulators.

testingbot espresso [appFile] [testAppFile] [options]

Arguments:

• appFile - Path to application APK file
• testAppFile - Path to test APK file containing Espresso tests

Device Options:

| Option | Description | |--------|-------------| | --app <path> | Path to application APK file | | --test-app <path> | Path to test APK file | | --device <name> | Device name (e.g., "Pixel 6", "Samsung.*") | | --platform-version <version> | Android OS version (e.g., "12", "13", "14") | | --real-device | Use a real device instead of an emulator | | --tablet-only | Only allocate tablet devices | | --phone-only | Only allocate phone devices | | --locale <locale> | Device locale (e.g., "en_US", "de_DE") | | --timezone <timezone> | Timezone (e.g., "America/New_York", "Europe/London") |

Test Configuration:

| Option | Description | |--------|-------------| | --name <name> | Test name for dashboard identification | | --build <build> | Build identifier for grouping test runs | | --test-runner <runner> | Custom test instrumentation runner | | --language <lang> | App language (ISO 639-1 code, e.g., "en", "fr", "de") |

Test Filtering:

| Option | Description | |--------|-------------| | --class <classes> | Run tests in specific classes (comma-separated fully qualified names) | | --not-class <classes> | Exclude tests in specific classes | | --package <packages> | Run tests in specific packages (comma-separated) | | --not-package <packages> | Exclude tests in specific packages | | --annotation <annotations> | Run tests with specific annotations (comma-separated) | | --not-annotation <annotations> | Exclude tests with specific annotations | | --size <sizes> | Run tests by size: small, medium, large (comma-separated) |

Network & Location:

| Option | Description | |--------|-------------| | --throttle-network <speed> | Network throttling: 4G, 3G, Edge, or airplane | | --geo-location <code> | Geographic IP location (ISO country code, e.g., "US", "DE") |

Tunnel:

| Option | Description | |--------|-------------| | -t, --tunnel | Start a TestingBot tunnel for this test run (cannot be combined with --async) | | --tunnel-identifier <id> | Identifier for the tunnel, allowing multiple tunnels in parallel |

Output Options:

| Option | Description | |--------|-------------| | --async | Start tests and exit without waiting for results | | -q, --quiet | Suppress progress output | | --report <format> | Download report after completion: html or junit | | --report-output-dir <path> | Directory to save reports (required with --report) |

Examples:

# Basic usage with positional arguments
testingbot espresso app.apk app-test.apk --device "Pixel 8"

# Using named options
testingbot espresso --app app.apk --test-app app-test.apk --device "Pixel 8"

# Real device with specific Android version
testingbot espresso app.apk app-test.apk \
  --device "Samsung Galaxy S24" \
  --platform-version "14" \
  --real-device

# Run specific test classes
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --class "com.example.LoginTest,com.example.HomeTest"

# Run tests with annotations
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --annotation "com.example.SmokeTest" \
  --size "small,medium"

# With network throttling and geolocation
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --throttle-network "3G" \
  --geo-location "DE" \
  --language "de"

# Download JUnit report
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --report junit \
  --report-output-dir ./reports

XCUITest

Run iOS XCUITest tests on real devices and simulators.

testingbot xcuitest [appFile] [testAppFile] [options]

Arguments:

• appFile - Path to application IPA file
• testAppFile - Path to test ZIP file containing XCUITests

Device Options:

| Option | Description | |--------|-------------| | --app <path> | Path to application IPA file | | --test-app <path> | Path to test ZIP file | | --device <name> | Device name (e.g., "iPhone 15", "iPad.*") | | --platform-version <version> | iOS version (e.g., "17.0", "18.2") | | --real-device | Use a real device instead of a simulator | | --tablet-only | Only allocate tablet devices | | --phone-only | Only allocate phone devices | | --orientation <orientation> | Screen orientation: PORTRAIT or LANDSCAPE | | --locale <locale> | Device locale (e.g., "DE", "US") | | --timezone <timezone> | Timezone (e.g., "America/New_York", "Europe/London") |

Test Configuration:

| Option | Description | |--------|-------------| | --name <name> | Test name for dashboard identification | | --build <build> | Build identifier for grouping test runs | | --language <lang> | App language (ISO 639-1 code, e.g., "en", "fr", "de") |

Network & Location:

| Option | Description | |--------|-------------| | --throttle-network <speed> | Network throttling: 4G, 3G, Edge, or airplane | | --geo-location <code> | Geographic IP location (ISO country code, e.g., "US", "DE") |

Tunnel:

| Option | Description | |--------|-------------| | -t, --tunnel | Start a TestingBot tunnel for this test run (cannot be combined with --async) | | --tunnel-identifier <id> | Identifier for the tunnel, allowing multiple tunnels in parallel |

Output Options:

| Option | Description | |--------|-------------| | --async | Start tests and exit without waiting for results | | -q, --quiet | Suppress progress output | | --report <format> | Download report after completion: html or junit | | --report-output-dir <path> | Directory to save reports (required with --report) |

Examples:

# Basic usage with positional arguments
testingbot xcuitest app.ipa app-test.zip --device "iPhone 16"

# Using named options
testingbot xcuitest --app app.ipa --test-app app-test.zip --device "iPhone 16"

# Real device with specific iOS version
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 15 Pro" \
  --platform-version "17.2" \
  --real-device

# iPad in landscape mode
testingbot xcuitest app.ipa app-test.zip \
  --device "iPad Pro" \
  --tablet-only \
  --orientation LANDSCAPE

# With localization settings
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --locale "DE" \
  --language "de" \
  --timezone "Europe/Berlin"

# With network throttling and geolocation
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --throttle-network "3G" \
  --geo-location "DE"

# Download HTML report
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --report html \
  --report-output-dir ./reports

# Run in background
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --async

Common Features

Real-time Progress

By default, the CLI shows real-time progress updates including:

• Test status updates with actual device names (even when using wildcards)
• Device allocation status
• Live output from Maestro flows

Use --quiet to suppress progress output.

Graceful Shutdown

Press Ctrl+C to gracefully stop running tests. The CLI will:

1. Stop all active test runs on TestingBot
2. Clean up resources
3. Exit with appropriate status code

Press Ctrl+C twice to force exit immediately.

Report Downloads

All test frameworks support downloading reports after completion:

# JUnit XML format (for CI integration)
--report junit --report-output-dir ./reports

# HTML format (for human viewing)
--report html --report-output-dir ./reports

Artifact Downloads (Maestro only)

Download all test artifacts including logs, screenshots, and video recordings:

testingbot maestro app.apk ./flows --download-artifacts --build "my-build"

Artifacts are saved as a zip file named after the --build value (or with a timestamp if not provided).

Exit Codes

• 0 - All tests passed
• 1 - One or more tests failed or an error occurred

Documentation

For more information, visit TestingBot Documentation.

License

MIT