tapctl

Mobile app control for coding agents.

tapctl is an AgentBrowser-style CLI for controlling real mobile devices, emulators, and simulators. It lets an LLM open your app, inspect the current screen, tap and fill elements by reference, navigate flows, and capture screenshots without writing an Appium or Playwright test first.

tapctl inspect
tapctl tap @e4
tapctl fill @e2 "[email protected]"
tapctl inspect

Use it from Codex, Claude Code, CI, or your own agent to:

• Explore and test an app while you build it
• Reproduce a bug across a real user flow
• Capture screenshots of interesting screens
• Verify that a feature works on a connected phone
• Give an agent interactive access to Android and iOS

Like AgentBrowser, the interface is built around ephemeral references. The agent observes the current screen, receives refs such as @e1, acts on them, then observes again. It does not need to invent coordinates or maintain a pre-authored test suite.

It is inspired by agent-browser and uses mobilecli's bundled Go binaries for the underlying Android and iOS device communication.

[!WARNING] This is an early alpha. Android real-device exploration works today, but commands and reference generation will change quickly.

The Agent Loop

Mobile testing frameworks are optimized for test authors. tapctl is optimized for an agent deciding what to do next:

1. Observe the current screen with inspect or snapshot.
2. Choose an element reference such as @e4.
3. Act with tap, fill, type, or press.
4. Observe again after the screen changes.

Installation

Requirements:

• Node.js 22 or newer
• A connected or booted Android/iOS device, emulator, or simulator
• USB debugging enabled and authorized when using a real Android device

Once published:

npm install -g tapctl

For now, install from source:

git clone <repository-url>
cd tapctl
npm install
npm run build
npm link

Verify the installation:

tapctl doctor
tapctl devices --include-offline

MobileCLI is bundled as an npm dependency. Users do not need to install or call it separately.

Use this checkout from other local projects

During development, globally link this checkout once:

npm install
npm run build
npm link

tapctl is then available on PATH from every project on the machine. After changing the TypeScript source, run npm run build; the global link keeps pointing at this checkout.

To install a fixed local copy instead of a live link:

npm install -g /absolute/path/to/tapctl

Agent Skill

Install the discovery skill with the open skills CLI:

npx skills add mike-grant/tapctl --skill tapctl

The discovery skill tells agents to load instructions matched to their installed CLI version:

tapctl skills get core

Quick Start

Select a connected device:

tapctl devices
tapctl use <device-id>

Inspect the foreground app:

tapctl inspect

inspect returns the foreground app, a screenshot artifact path, and actionable references:

{
  "deviceId": "device-id",
  "foreground": {
    "status": "ok",
    "data": {
      "packageName": "com.example.app"
    }
  },
  "screenshot": "/project/.tapctl/artifacts/screenshot-123.png",
  "refs": [
    {
      "ref": "@e1",
      "label": "Sign in",
      "role": "ViewGroup",
      "bounds": {
        "x": 40,
        "y": 500,
        "width": 300,
        "height": 80
      }
    }
  ]
}

Act on a reference, then inspect again:

tapctl tap @e1
tapctl inspect

Common Workflows

Launch and explore an app

tapctl launch com.example.app
tapctl inspect
tapctl tap @e4
tapctl inspect

Fill a form

tapctl snapshot
tapctl fill @e2 "[email protected]"
tapctl fill @e3 "password"
tapctl tap @e4
tapctl wait --text "Welcome"
tapctl inspect

Capture screenshots

tapctl screenshot
tapctl screenshot ./artifacts/settings.png

Debug missing references

tapctl snapshot --all
tapctl dump

Use coordinates only when the UI tree does not expose a useful element:

tapctl tap 100,200

Commands

tapctl start                         Start the MobileCLI JSON-RPC server
tapctl doctor                        Diagnose backend and device access
tapctl skills list                   List bundled runtime skills
tapctl skills get core               Print version-matched agent guidance
tapctl devices [--include-offline]   List devices
tapctl use <device-id>               Persist the default device
tapctl launch <bundle-or-package>    Launch an app
tapctl foreground                    Show the foreground app
tapctl inspect [-i|--all]            Screenshot, foreground app, and refs
tapctl snapshot [-i|--all] [--json]  Create and persist element refs
tapctl dump                          Print the raw UI tree
tapctl tap <@ref|x,y>                Tap a reference or coordinates
tapctl fill <@ref> <text>            Tap a reference and input text
tapctl type <text>                   Input text into the focused field
tapctl press <button>                Press BACK, HOME, ENTER, etc.
tapctl screenshot [path]             Save a screenshot artifact
tapctl wait --text <text>            Wait for text to appear
tapctl state                         Print the persisted session

Run tapctl help for the current command list.

How It Works

tapctl is a TypeScript orchestration layer over MobileCLI:

LLM or coding agent
        |
        v
tapctl TypeScript CLI
  - session state
  - screenshots and artifacts
  - UI snapshot normalization
  - ephemeral @e references
        |
        v
MobileCLI Go binary
  - Android Debug Bridge
  - iOS simulators and device agents
  - screenshots, UI trees, and input
        |
        v
Android / iOS device

The CLI first attempts to use MobileCLI's local JSON-RPC server and falls back to invoking its bundled MobileCLI backend. Session state and generated artifacts are stored in .tapctl/ in the current working directory.

References belong to the latest snapshot. Re-run inspect or snapshot after navigation, modal changes, keyboard changes, or other meaningful UI updates.

Platform Status

| Platform | Status | | --- | --- | | Android real devices | Tested | | Android emulators | Expected to work through MobileCLI | | iOS simulators | Expected to work through MobileCLI | | iOS real devices | Requires MobileCLI's provisioned on-device agent |

Current limitations:

• Reference generation is heuristic and may include non-actionable elements.
• Some container references do not have an ideal tap point.
• Devices can sleep, lock, or surface system UI during a flow.
• There is not yet a packaged agent skill.
• Automated tests and CI have not been added yet.

Development

npm install
npm run check
npm test
npm run build
npm link

The project is intentionally small while the interaction model is being proven. The main implementation is in src/cli.ts.

Roadmap

• Improve actionable-element detection and hit-point selection
• Add swipe, long press, app lifecycle, and device-management commands
• Add screenshot annotation and state diffs
• Expand automated device integration testing

License

MIT