twiddle-mcp

Stop your agent twiddling its thumbs. — a Twiddle Thumb Studio tool

Blocking wait_for primitives and semantic UI diffs for the iOS Simulator and Android Emulator. An MCP server that lets coding agents (Claude Code, Cursor, any MCP client) drive and — critically — verify mobile apps without screenshot → sleep → screenshot polling.

Web agents already have this: Playwright MCP's accessibility snapshots and browser_wait_for. Mobile agents don't. This fills that gap.

agent: tap(ref=e4)                      → "Tapped button 'Load Data'"
agent: wait_for({text: "Welcome back"}) → blocks…
                                        → MATCHED in 1452ms:
                                          ~ text #statusLabel "Loading…" → "Welcome back, Tyler"

One tool call. No sleeps, no screenshot diffing, no wasted tokens.

Setup

// .mcp.json
{
  "mcpServers": {
    "twiddle": { "command": "npx", "args": ["-y", "@twiddlethumb/twiddle-mcp"] }
  }
}

Requirements: macOS with Xcode (iOS) and/or the Android SDK with adb on PATH. First iOS use compiles WebDriverAgent (~30s–3min, then cached). Android bootstraps in ~3s.

Tools

Verify primitives (the point of this project):

| Tool | What it does | |---|---| | wait_for | Block until text/element appears (or state:"gone"). Returns elapsed ms + semantic diff. On timeout returns matched:false with the current tree — never throws. | | wait_for_idle | Block until the UI stops changing. Volatile elements (clocks, spinners) are auto-detected and ignored. | | get_ui_tree | Accessibility tree as text with [ref=eN] markers — cheaper and more precise than a screenshot. | | ui_diff | "What changed since snapshot N?" as short sentences. |

Driver: list_devices · launch_app · terminate_app · tap (by ref / selector / x,y — refs re-resolve against the live tree; a vanished element returns the current tree instead of a blind tap) · type_text · swipe · press_button · screenshot (fallback for purely visual checks).

How it works

Both platforms expose a localhost HTTP automation server — WebDriverAgent on iOS (bootstrapped from the appium-webdriveragent npm package), appium-uiautomator2-server on Android (APKs installed over adb). One keep-alive connection pool and one shared poll loop per device (~7Hz) serve any number of concurrent waits. Trees are normalized to a platform-agnostic node model; elements get stable refs via identity keys (role+identifier first), so refs survive re-renders; diffs align by those keys and suppress frame jitter and volatile elements.

Development

pnpm install
pnpm test        # unit tests (tree engine, wait logic) — no devices needed
pnpm build       # dist/index.js
examples/ios-demo/build.sh   # build + install the SwiftUI demo on the booted sim

Benchmark

Measured against mobile-mcp, Maestro MCP, and the raw screenshot baseline on both platforms (tap → verify a 2s async load; medians over repeated runs, canonical agent patterns, full method in bench/RESULTS.md):

| | agent turns | context/verification | est. total (Android / iOS) | |---|---|---|---| | twiddle-mcp | 2 | 1 KB | 11s / 9s | | mobile-mcp | 3–4 (1 poll/turn) | 3 KB | 16s / 14s* | | maestro-mcp | 1 (flow YAML) | 0.3 KB | 10s / 6s | | screenshots + shell (default) | 3 | 117 KB | 12s / ✗ (simctl cannot tap) |

* iOS requires a workaround for mobile-mcp's fractional-coordinate click bug (see RESULTS.md).

On failure (wait_for timeout), the agent gets the full current UI tree to reason over; a failed Maestro flow returns a 288-byte error string, and polling patterns return their last snapshot. Every extra second of app latency costs polling tools another turn; it costs wait_for nothing.

Status

Early but working: iOS + Android verified end-to-end over real MCP stdio (see bench/ and examples/). Roadmap: event-driven AccessibilityService transport on Android, iOS benchmark, agent-level benchmark via headless Claude Code sessions (bench/run.mjs).

MIT