GitHub - testingbot/testingbotctl: A command line interface for the TestingBot platform. Run Maestro, XCUITest and Espresso.

Run mobile tests on real devices in the cloud

Run Espresso, XCUITest and Maestro tests on real devices in the cloud.

Real Devices — Test on thousands of real iOS and Android devices
Emulators & Simulators — Fast feedback with virtual devices
Parallel Execution — Split tests across multiple devices with sharding
CI/CD Ready — Integrates with GitHub Actions, Jenkins, and more
Live Results — Watch tests run in real-time
Artifacts — Download videos, screenshots, and logs

Installation

npm install -g @testingbot/cli

Requirements: NodeJS 20 or higher

Authentication

The CLI requires TestingBot API credentials. You can authenticate in several ways:

Browser Login (Recommended)

This opens your browser for authentication. After logging in, your credentials are saved to ~/.testingbot.

Other Methods

Command-line options: --api-key and --api-secret
Environment variables: TB_KEY and TB_SECRET
Config file: Create ~/.testingbot with content key:secret

Commands

Maestro

Run Maestro UI tests on real devices and emulators/simulators.

testingbot maestro <app> <flows...> [options]

Arguments:

app - Path to your app file (.apk, .ipa, .app, or .zip)
flows - One or more paths to flow files (.yaml/.yml), directories, .zip files, or glob patterns

App Options:

Option	Description
`--app <path>`	Path to the application under test (alternative to the positional `app` argument)
`--other-app <path-or-url>`	Additional companion app to install on the device alongside `--app`. Accepts a local file path (`.apk`, `.ipa`, `.app`, `.zip`) or a `tb://<appkey>` / `http(s)://...` URL — local paths are uploaded; URLs are passed through to the run as-is. Repeatable, max 4 entries.

Device Options:

Option	Description
`--device <name>`	Device name (e.g., "Pixel 9", "iPhone 16")
`--platform <name>`	Platform: Android or iOS
`--deviceVersion <version>`	OS version (e.g., "14", "17.2")
`--real-device`	Use a real device instead of emulator/simulator
`--orientation <orientation>`	Screen orientation: PORTRAIT or LANDSCAPE
`--device-locale <locale>`	Device locale (e.g., "en_US", "de_DE")
`--timezone <timezone>`	Timezone (e.g., "America/New_York", "Europe/London")

Test Configuration:

Option	Description
`--name <name>`	Test name for dashboard identification
`--build <build>`	Build identifier for grouping test runs
`--groups <names>`	Tag the test session with one or more groups (comma-separated). Groups appear on the test in the TestingBot dashboard
`--include-tags <tags>`	Only run flows with these tags (comma-separated)
`--exclude-tags <tags>`	Exclude flows with these tags (comma-separated)
`-e, --env <KEY=VALUE>`	Environment variable for flows (can be repeated)
`--config <path>`	Path to a custom Maestro config file (default: config.yaml in project root)
`--maestro-version <version>`	Maestro version to use (e.g., "2.0.10")

Network & Location:

Option	Description
`--throttle-network <speed>`	Network throttling: 4G, 3G, Edge, airplane, or disable
`--geo-country-code <code>`	Geographic IP location (ISO country code, e.g., "US", "DE")

Tunnel:

Option	Description
`-t, --tunnel`	Start a TestingBot tunnel for this test run (cannot be combined with `--async`)
`--tunnel-identifier <id>`	Identifier for the tunnel, allowing multiple tunnels in parallel

Output Options:

Option	Description
`--async`	Start tests and exit without waiting for results
`-q, --quiet`	Suppress progress output
`--report <format>`	Download report after completion: html or junit
`--report-output-dir <path>`	Directory to save reports (required with --report)
`--download-artifacts [mode]`	Download test artifacts (logs, screenshots, video). Mode: `all` (default) or `failed`
`--artifacts-output-dir <path>`	Directory to save artifacts zip (defaults to current directory)

Advanced Options:

Option	Description
`--shard-split <number>`	Split flows into N parallel sessions for faster execution
`--retry <count>`	Retry failed flows up to N times (0-2, default 0). Re-runs only the flows (or shards) that failed, the moment they fail, while the rest of the run continues. Cannot be combined with `--async`.
`--ignore-checksum-check`	Skip checksum verification and always upload the app

Note on --retry: a failed flow/shard is retried immediately — as soon as it fails — without waiting for the other flows in the run to finish. Retry attempts appear live in the flow table marked with a ↻ icon. Each flow is retried independently up to N times, stopping as soon as that flow passes. Pass/fail uses the result of the last attempt per flow (last-attempt-wins), consistently across the CLI exit code, the TestingBot dashboard, and reports downloaded via --report.

CI/CD Integration:

Option	Description
`--commit-sha <sha>`	Git commit SHA associated with this test run
`--pull-request-id <id>`	Pull request ID this test run originated from
`--repo-name <name>`	Repository name (e.g., GitHub repo slug)
`--repo-owner <owner>`	Repository owner (e.g., GitHub organization or username)

Examples:

# Basic usage
testingbot maestro app.apk ./flows

# Multiple flow directories
testingbot maestro app.apk ./flows/smoke ./flows/regression ./flows/e2e

# With device selection
testingbot maestro app.apk ./flows --device "Pixel 8" --deviceVersion "14"

# Android app on real device with tags
testingbot maestro app.apk ./flows --device "Samsung Galaxy S24" --real-device --include-tags "smoke,regression"

# Tag the test session with groups (visible in the dashboard)
testingbot maestro app.apk ./flows --groups "smoke,critical"

# With environment variables
testingbot maestro app.apk ./flows -e API_URL=https://staging.example.com -e API_KEY=secret

# With companion apps installed alongside the main app (up to 4)
# Each --other-app can be a local file (uploaded) or a tb:// / http(s):// URL (passed through)
testingbot maestro --app main.apk \
  --other-app helper.apk \
  --other-app tb://existing-appkey \
  --other-app https://example.com/mock-server.apk \
  ./flows

# Download JUnit report
testingbot maestro app.apk ./flows --report junit --report-output-dir ./reports

# Download all artifacts (logs, screenshots, video)
testingbot maestro app.apk ./flows --download-artifacts --build "build-123"

# Download artifacts only for failed tests
testingbot maestro app.apk ./flows --download-artifacts failed --artifacts-output-dir ./artifacts

# Use a custom config file
testingbot maestro app.apk ./flows --config .maestro/ci-config.yaml

# Run in background (async)
testingbot maestro app.apk ./flows --async

# Split flows across 3 shards, grouping all flows over 3 parallel sessions
testingbot maestro app.apk ./flows --shard-split 3

# Retry failed flows up to 2 times (re-runs only the flows that failed)
testingbot maestro app.apk ./flows --retry 2

# CI/CD integration with Git metadata
testingbot maestro app.apk ./flows \
  --commit-sha "abc123def" \
  --pull-request-id "42" \
  --repo-owner "myorg" \
  --repo-name "myapp"

Espresso

Run Android Espresso tests on real devices and emulators.

testingbot espresso [appFile] [testAppFile] [options]

Arguments:

appFile - Path to application APK file
testAppFile - Path to test APK file containing Espresso tests

Device Options:

Option	Description
`--app <path>`	Path to application APK file
`--test-app <path>`	Path to test APK file
`--device <name>`	Device name (e.g., "Pixel 6", "Samsung.*")
`--platform-version <version>`	Android OS version (e.g., "12", "13", "14")
`--real-device`	Use a real device instead of an emulator
`--tablet-only`	Only allocate tablet devices
`--phone-only`	Only allocate phone devices
`--locale <locale>`	Device locale (e.g., "en_US", "de_DE")
`--timezone <timezone>`	Timezone (e.g., "America/New_York", "Europe/London")

Test Configuration:

Option	Description
`--name <name>`	Test name for dashboard identification
`--build <build>`	Build identifier for grouping test runs
`--test-runner <runner>`	Custom test instrumentation runner
`--language <lang>`	App language (ISO 639-1 code, e.g., "en", "fr", "de")

Test Filtering:

Option	Description
`--class <classes>`	Run tests in specific classes (comma-separated fully qualified names)
`--not-class <classes>`	Exclude tests in specific classes
`--package <packages>`	Run tests in specific packages (comma-separated)
`--not-package <packages>`	Exclude tests in specific packages
`--annotation <annotations>`	Run tests with specific annotations (comma-separated)
`--not-annotation <annotations>`	Exclude tests with specific annotations
`--size <sizes>`	Run tests by size: small, medium, large (comma-separated)

Network & Location:

Option	Description
`--throttle-network <speed>`	Network throttling: 4G, 3G, Edge, or airplane
`--geo-location <code>`	Geographic IP location (ISO country code, e.g., "US", "DE")

Tunnel:

Option	Description
`-t, --tunnel`	Start a TestingBot tunnel for this test run (cannot be combined with `--async`)
`--tunnel-identifier <id>`	Identifier for the tunnel, allowing multiple tunnels in parallel

Output Options:

Option	Description
`--async`	Start tests and exit without waiting for results
`-q, --quiet`	Suppress progress output
`--report <format>`	Download report after completion: html or junit
`--report-output-dir <path>`	Directory to save reports (required with --report)

Examples:

# Basic usage with positional arguments
testingbot espresso app.apk app-test.apk --device "Pixel 8"

# Using named options
testingbot espresso --app app.apk --test-app app-test.apk --device "Pixel 8"

# Real device with specific Android version
testingbot espresso app.apk app-test.apk \
  --device "Samsung Galaxy S24" \
  --platform-version "14" \
  --real-device

# Run specific test classes
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --class "com.example.LoginTest,com.example.HomeTest"

# Run tests with annotations
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --annotation "com.example.SmokeTest" \
  --size "small,medium"

# With network throttling and geolocation
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --throttle-network "3G" \
  --geo-location "DE" \
  --language "de"

# Download JUnit report
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --report junit \
  --report-output-dir ./reports

XCUITest

Run iOS XCUITest tests on real devices and simulators.

testingbot xcuitest [appFile] [testAppFile] [options]

Arguments:

appFile - Path to application IPA file
testAppFile - Path to test ZIP file containing XCUITests

Device Options:

Option	Description
`--app <path>`	Path to application IPA file
`--test-app <path>`	Path to test ZIP file
`--device <name>`	Device name (e.g., "iPhone 15", "iPad.*")
`--platform-version <version>`	iOS version (e.g., "17.0", "18.2")
`--real-device`	Use a real device instead of a simulator
`--tablet-only`	Only allocate tablet devices
`--phone-only`	Only allocate phone devices
`--orientation <orientation>`	Screen orientation: PORTRAIT or LANDSCAPE
`--locale <locale>`	Device locale (e.g., "DE", "US")
`--timezone <timezone>`	Timezone (e.g., "America/New_York", "Europe/London")

Test Configuration:

Option	Description
`--name <name>`	Test name for dashboard identification
`--build <build>`	Build identifier for grouping test runs
`--language <lang>`	App language (ISO 639-1 code, e.g., "en", "fr", "de")

Network & Location:

Option	Description
`--throttle-network <speed>`	Network throttling: 4G, 3G, Edge, or airplane
`--geo-location <code>`	Geographic IP location (ISO country code, e.g., "US", "DE")

Tunnel:

Option	Description
`-t, --tunnel`	Start a TestingBot tunnel for this test run (cannot be combined with `--async`)
`--tunnel-identifier <id>`	Identifier for the tunnel, allowing multiple tunnels in parallel

Output Options:

Option	Description
`--async`	Start tests and exit without waiting for results
`-q, --quiet`	Suppress progress output
`--report <format>`	Download report after completion: html or junit
`--report-output-dir <path>`	Directory to save reports (required with --report)

Examples:

# Basic usage with positional arguments
testingbot xcuitest app.ipa app-test.zip --device "iPhone 16"

# Using named options
testingbot xcuitest --app app.ipa --test-app app-test.zip --device "iPhone 16"

# Real device with specific iOS version
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 15 Pro" \
  --platform-version "17.2" \
  --real-device

# iPad in landscape mode
testingbot xcuitest app.ipa app-test.zip \
  --device "iPad Pro" \
  --tablet-only \
  --orientation LANDSCAPE

# With localization settings
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --locale "DE" \
  --language "de" \
  --timezone "Europe/Berlin"

# With network throttling and geolocation
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --throttle-network "3G" \
  --geo-location "DE"

# Download HTML report
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --report html \
  --report-output-dir ./reports

# Run in background
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --async

Common Features

Real-time Progress

By default, the CLI shows real-time progress updates including:

Test status updates with actual device names (even when using wildcards)
Device allocation status
Live output from Maestro flows

Use --quiet to suppress progress output.

Graceful Shutdown

Press Ctrl+C to gracefully stop running tests. The CLI will:

Stop all active test runs on TestingBot
Clean up resources
Exit with appropriate status code

Press Ctrl+C twice to force exit immediately.

Report Downloads

All test frameworks support downloading reports after completion:

# JUnit XML format (for CI integration)
--report junit --report-output-dir ./reports

# HTML format (for human viewing)
--report html --report-output-dir ./reports

Artifact Downloads (Maestro only)

Download all test artifacts including logs, screenshots, and video recordings:

testingbot maestro app.apk ./flows --download-artifacts --build "my-build"

Artifacts are saved as a zip file named after the --build value (or with a timestamp if not provided).

Exit Codes

0 - All tests passed
1 - One or more tests failed or an error occurred

Documentation

For more information, visit TestingBot Documentation.

License

MIT