pi-computer-use

Pi extension for GUI computer-use on macOS. Gives your agent eyes and hands — it can see the screen, find UI elements, and interact with any app through native mouse/keyboard events.

Useful for launching, testing, and debugging GUI applications from pi.

How it works

1. Screenshot — captures the screen or app window via macOS screencapture
2. Grounding — sends the screenshot + a target description (e.g. 'button labeled "Save"') to a vision model to get pixel coordinates
3. Action — dispatches native input events via a compiled Swift helper

The Swift binary is compiled on first use and cached. No manual build step needed.

Install

pi install git:github.com/swairshah/pi-computer-use

The extension uses a Swift native helper for mouse/keyboard events, compiled automatically on first use. You'll need:

• Xcode Command Line Tools — xcode-select --install if you don't have them
• Accessibility permission for your terminal (System Settings → Privacy & Security → Accessibility)
• Screen Recording permission for your terminal (System Settings → Privacy & Security → Screen Recording)

Tools

Observation

| Tool | What it does | |------|-------------| | gui_read | Screenshot + optionally locate a target element | | gui_screenshot | Screenshot only | | gui_cursor_position | Current mouse (x, y) | | gui_clipboard_read | Read system clipboard |

Mouse

| Tool | What it does | |------|-------------| | gui_click | Left/right/middle click. Supports modifier keys (Shift+click, Cmd+click, etc.) | | gui_double_click | Double-click (select word, open file) | | gui_triple_click | Triple-click (select line/paragraph) | | gui_right_click | Right-click (context menu) | | gui_hover | Hover (tooltips, hover menus) | | gui_drag | Drag from A to B. Supports modifiers (Option+drag to duplicate) | | gui_scroll | Scroll up/down/left/right |

Keyboard

| Tool | What it does | |------|-------------| | gui_type | Type text into a field (optionally click target first) | | gui_keypress | Press a key (Enter, Tab, Escape, arrows, etc.) | | gui_hotkey | Keyboard shortcut (Cmd+S, Shift+Cmd+P, etc.) |

Utility

| Tool | What it does | |------|-------------| | gui_clipboard_write | Write to system clipboard | | gui_wait | Pause N milliseconds (animations, loading) | | gui_batch | Chain multiple actions in one tool call |

gui_batch

Executes a sequence of actions without round-tripping through the LLM between each step. Each grounded action (click, type with target) takes a fresh screenshot, but you save inference calls.

gui_batch({ actions: [
  { action: "click", target: "search field" },
  { action: "type", value: "hello world" },
  { action: "keypress", key: "Enter" },
  { action: "wait", ms: 1000 },
  { action: "scroll", direction: "down", amount: 10 }
]})

Supported actions: click, right_click, double_click, triple_click, hover, drag, scroll, type, keypress, hotkey, wait, clipboard_read, clipboard_write. Stops on first error.

Source

src/
├── index.ts          # Extension entry — registers tools with pi
├── runtime.ts        # Screenshot capture, grounding, native input dispatch
├── grounding.ts      # Vision model grounding (uses pi's model registry + pi-ai)
├── native-helper.ts  # Embedded Swift source, compiled and cached at runtime
└── learn.ts          # /learn command — record GUI demos and save as skills

Credits

GUI runtime adapted from understudy.