mcp-mobile-interaction

An MCP (Model Context Protocol) server that lets Claude interact with Android and iOS devices/emulators. Take screenshots, tap, swipe, type, inspect UI elements, mock GPS, record the screen, and more — no Appium required.

Prerequisites

Android

• Android SDK with adb in your PATH
• An Android emulator running or a physical device connected via USB with ADB debugging enabled

iOS

• macOS with Xcode installed (provides xcrun simctl) — covers screenshots, app lifecycle, clipboard, location, appearance and recording on simulators
• idb (brew install idb-companion && pip install fb-idb) — required for all UI interaction (tap, swipe, type, key presses, UI tree) on simulators AND physical devices. xcrun simctl has no UI interaction commands.

Run the doctor tool to diagnose your setup.

Installation

With Claude Code

claude mcp add mobile -- npx -y mcp-mobile-interaction

Or add .mcp.json to your project root (shared with your team):

{
  "mcpServers": {
    "mobile": {
      "command": "npx",
      "args": ["-y", "mcp-mobile-interaction"]
    }
  }
}

With Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "mobile": {
      "command": "npx",
      "args": ["-y", "mcp-mobile-interaction"]
    }
  }
}

Manual

npm install -g mcp-mobile-interaction

Tools

All tools accept a platform parameter ("android" or "ios") and an optional device_id (defaults to the first connected device). Read-only tools are annotated with readOnlyHint so MCP hosts can auto-approve them.

Inspection Tools

| Tool | Description | |------|-------------| | list_devices | List connected devices and emulators/simulators | | screenshot | Capture a screenshot (base64 JPEG). Supports cropping to a single element (crop_resource_id / crop_text) for token-efficient component checks | | get_ui_tree | Compact flat list of UI elements with optional filters (only_clickable, only_with_text, type_filter, resource_id_contains, max_elements) | | get_screen_info | Screen dimensions, density, and orientation (rotation-aware on Android) | | get_screen_state | UI tree + screenshot in a single call (saves a round-trip) | | find_element | Find elements by text/resource_id/type without tapping. For assertions. Supports scroll_to_find | | get_current_app | Foreground app package + activity (Android). For asserting navigation/deep links | | get_app_info | Whether an app is installed + its version | | get_device_logs | OS-level logs (Android logcat / iOS log show on simulators). Filter by tag, level, or search string | | get_clipboard | Read the device clipboard (verify copy-to-clipboard features) | | doctor | Diagnose local tooling: adb, ANDROID_HOME, devices, emulator, simctl, idb |

Action Tools

| Tool | Description | |------|-------------| | tap / double_tap / long_press | Touch at (x, y) coordinates (native resolution) | | swipe | Swipe between coordinates or by direction (up/down/left/right) | | tap_element | Find element by text/resource_id/type and tap it. Supports scroll_to_find (+scroll_direction) and wait_for. Warns when the target is disabled or covered by an overlay | | type_text | Type into the focused input. Full Unicode: non-ASCII text (á, ñ, emoji) is delivered via clipboard paste on Android — adb input text silently drops it | | clear_text | Clear the focused text field (reads its length from the UI tree on Android) | | press_key | Press a named key (incl. paste) or raw Android keycode, with repeat support | | launch_app / kill_app | Start / force-stop an app | | install_app / uninstall_app | Install a local .apk / .app / .ipa, or remove an app | | open_url | Open a URL or deep link (query params with & are quoted correctly) | | clear_app_data | Mode cache clears temp files only; mode all resets to fresh install | | set_clipboard | Set the device clipboard (targets the simulator pasteboard on iOS, not the host Mac) | | set_location | Mock GPS coordinates (Android emulator / iOS simulator+idb). For delivery/route flows | | set_network_state | Wi-Fi, mobile data, airplane mode, and emulator latency/speed throttling (Android) | | set_appearance | Switch dark/light mode | | rotate_device | Rotate to a fixed orientation (Android) | | record_screen | Start/stop an mp4 screen recording — bug repro evidence |

Waiting Tools

| Tool | Description | |------|-------------| | wait_for_element | Poll until an element matching text/type/resource_id criteria appears | | wait_for_element_gone | Poll until a matching element disappears (spinners, skeletons, dialogs) | | wait_for_stable | Poll until the screen stops changing (two consecutive UI snapshots match) |

UI Tree Format

UI trees are returned in a compact one-line-per-element format (~4x fewer tokens than JSON):

UI tree (12; format: [n] Type "text" @(center_x,center_y) WxH #resource_id flags):
[0] TextView "Settings" @(270,125) 540x50 #title clickable
[1] EditText "" @(540,300) 900x120 #search_input focused
[2] Button "Save" @(540,960) 300x90 #save_btn disabled

Flags: clickable, disabled, focused, overlay (an element that looks like a modal scrim). Output is capped (default 120 elements) with a summary line pointing to the filters.

Coordinate System

Screenshots are scaled down by default (scale=0.5) to save bandwidth, while get_ui_tree and all coordinate-based tools (tap, double_tap, long_press, swipe) work in native device resolution.

Every screenshot response includes the native dimensions and scale factor to make this explicit:

Screenshot captured (540x1140, scale=0.5 of native 1080x2280).
Coordinate tools expect native resolution — multiply screenshot pixel
positions by 2 to convert, or pass screenshot_scale=0.5.

Two ways to handle this:

1. Manual conversion — multiply the position you see in the screenshot by 1/scale (e.g. ×2 for scale=0.5)
2. Automatic conversion — pass screenshot_scale to coordinate tools and they convert for you:

tap(x=270, y=570, screenshot_scale=0.5)
→ taps at native (540, 1140)

The screenshot_scale parameter is available on tap, double_tap, long_press, and swipe.

Observe Mode

Action tools (tap, double_tap, long_press, swipe, type_text, press_key, launch_app, open_url, tap_element, clear_text) support optional observe parameters that capture the screen state after the action completes — returning the result in a single round-trip instead of two:

| Parameter | Description | |-----------|-------------| | observe | "none" (default), "ui_tree", "screenshot", or "both" | | observe_delay_ms | Milliseconds to wait before capturing (default: 500) | | observe_stabilize | If true, wait for UI to stop changing instead of a fixed delay |

Example: before vs after

Before (2 calls):

tap(x=540, y=960) → get_ui_tree()

After (1 call):

tap(x=540, y=960, observe="ui_tree")

For a 5-step test flow, this cuts round-trips roughly in half.

Examples

Take a screenshot

"Take a screenshot of my Android emulator"

Navigate an app

"Open Settings on my iOS simulator, then scroll down and tap General"

Claude will use launch_app and tap_element with scroll_to_find: true.

Test a delivery route with mock GPS

"Set the location to the first stop of the route and verify the app shows 'You have arrived'"

Claude will use set_location, then wait_for_element.

Record a bug repro

"Record the screen while you reproduce the crash, then give me the video"

Claude will use record_screen (start), drive the flow, then record_screen (stop) and return the mp4 path.

Verify copy-to-clipboard

"Tap the copy tracking code button and verify the clipboard contains the code"

Claude will use tap_element, then get_clipboard.

Type Spanish text

"Fill the notes field with 'Entregar mañana según lo acordado'"

type_text detects the non-ASCII characters and delivers them via clipboard paste — adb shell input text would silently drop them.

How It Works

• Android: Uses adb directly (screencap, input, uiautomator, am, pm, dumpsys, emu console). Commands run via execFile (no shell), with device-side quoting where needed — text and URLs with special characters are safe.
• iOS Simulators: Uses xcrun simctl for lifecycle/screenshots/clipboard/location/appearance/recording, and idb for all UI interaction (simctl has no tap/swipe/type).
• iOS Physical Devices: Uses idb (Facebook's iOS Development Bridge).

Screenshots are compressed with sharp (resized + JPEG quality) to stay under Claude's 1MB image limit.

Development

mise install   # pins Node 22
npm install
npm test
npm run build

License

MIT