@amaster.ai/pi-computer-use

Cross-platform computer-use tools for Pi desktop automation. The extension exposes a native MCP tool surface with a computer_use_ prefix. The bundled runtime comes from the official Cua Driver Rust 0.9.0 release.

What it provides

• One Rust 0.9.0 driver line across macOS, Linux, and Windows
• 49 version-pinned upstream tools, including sessions, element tokens, accessibility + screenshot state, native input, browser tools, diagnostics, recording, and permission policy support
• Full MCP text, image, and structuredContent forwarding
• Owned daemon + MCP proxy lifecycle with session-owned reconnect and per-call cancellation
• A non-prompting Linux/Windows permission probe on session start
• Once-per-session app-launch approval and confirmation for high-risk operations
• Bounded text and structured results before they enter Pi's context
• Optional secondary vision analysis through a configured Pi model

get_window_state is the primary perception tool. Cua Driver 0.9 returns the accessibility tree, structured elements with element_token, and a screenshot in one response. The standalone screenshot tool no longer exists.

Install

bun add @amaster.ai/pi-computer-use

The package bundles signed/precompiled driver assets. No separate Cua Driver installation is required.

Configuration

Configure .pi/settings.json or ~/.pi/agent/settings.json:

Project settings are loaded only after project trust is accepted. ${ENV_VAR} interpolation is supported in user and agent settings, but not in project settings.

{
  "pi-computer-use": {
    "mode": "bundled",
    "confirmAppLaunch": true,
    "confirmDangerousActions": true
  }
}

| Option | Type | Default | Description | | --- | --- | --- | --- | | mode | "bundled" \| "path" | "bundled" | Use the packaged 0.9.0 driver or a compatible custom binary | | binaryPath | string | — | Custom driver path when mode is "path" | | extraArgs | string[] | — | Additional arguments appended to cua-driver mcp | | confirmAppLaunch | boolean | true | Ask once per app target before launch_app | | confirmDangerousActions | boolean | true | Confirm high-risk tools such as kill_app and replay_trajectory; recording always requires confirmation | | visionModel | { provider, model } | — | Register computer_use_analyze_screenshot |

In non-interactive modes, confirmation-required tools return an error unless the corresponding confirmation setting is explicitly disabled.

Optional vision model

{
  "pi-computer-use": {
    "visionModel": {
      "provider": "openai",
      "model": "gpt-4o"
    }
  }
}

computer_use_analyze_screenshot requires both pid and window_id. It calls get_window_state, reuses the returned image, and invokes the configured model. Use it only when the primary model cannot resolve visual ambiguity.

Runtime and permissions

On macOS, session_start registers the generated 0.9.0 manifest without starting the driver. The signed app and MCP proxy start lazily on the first computer-use tool call, which requests any missing permissions through check_permissions({ prompt: true }). Existing grants do not raise another system dialog. The requested tool still runs and reports its own capability or permission error. Linux and Windows keep eager startup: they discover the exact live tools/list surface and call check_permissions({ prompt: false }). If discovery fails, the extension registers computer_use_connect (and /computer-use-connect) so a later retry can install the exact live platform contract without advertising another OS's schemas.

Driver startup, reconnect, and the first macOS permission probe are session-owned. Cancelling a tool stops only that caller's wait or MCP request; session shutdown aborts the shared work. macOS and Linux also use a transient pipe-backed lease to stop the owned daemon after an abrupt host exit; no persistent service or system scheduler is installed.

• Bundled macOS: launches the signed CuaDriver.app through LaunchServices, so Accessibility and Screen Recording grants belong to com.trycua.driver.
• Custom macOS binary: uses Cua Driver embedded mode and inherits the host application's TCC responsibility chain.
• Linux/Windows: starts an extension-owned daemon in the interactive user session and tears it down on session shutdown.

Supported targets

| Platform | Bundled target | | --- | --- | | macOS ARM64 / x64 | bin/darwin-universal/CuaDriver.app | | Linux x64 | bin/linux-x64/cua-driver | | Linux ARM64 | bin/linux-arm64/cua-driver | | Windows x64 | bin/win32-x64/cua-driver.exe + cua-driver-uia.exe | | Windows ARM64 | bin/win32-arm64/cua-driver.exe + cua-driver-uia.exe |

Canonical workflow

1. computer_use_start_session
2. computer_use_launch_app or computer_use_list_windows
3. computer_use_get_window_state
4. Act using element_token/element_index, falling back to pixels for custom-drawn surfaces
5. Re-run computer_use_get_window_state and verify the change
6. computer_use_end_session

Linux and Windows tool descriptions and schemas come from the exact live driver. macOS uses the generated manifest for the bundled driver release.

License

Apache-2.0 for this package. Bundled Cua Driver assets retain their upstream license and release metadata.