pi-automation

Automation toolkit for the pi coding agent. Aggregates web access, browser control, screen/window tools, and native Windows UI automation into a single extension.

Tools

| Tool | Source | Description | |------|--------|-------------| | web_search | pi-web-access | Web search (multiple providers) | | source_check | pi-web-access | Verify a claim with source citations | | fetch_content | pi-web-access | Fetch URLs (GitHub / PDF / video / YouTube) | | get_search_content | pi-web-access | Retrieve stored search/fetch content | | browser | local | Drive a real browser via CDP (launch/list/navigate/click/type/js/read/wait/screenshot/close) | | take_screenshot | local | Capture screen (single/stitched/region/scale) | | hide_window | local | Hide / show a window | | ui_automation | local | Read & operate native Windows UI controls |

Commands (from pi-web-access): /websearch, /curator, /google-account, /search. Shortcuts: Ctrl+Shift+S (curate), Ctrl+Shift+W (activity).

ui_automation actions

Target a window via process (name/PID), window_title, or window_handle (hex). Match controls by name, automation_id, or control_type.

Control actions

• list_windows — list top-level windows
• list_controls — enumerate a window's control tree
• find — find controls by name / controlType / automationId
• click — invoke a control (InvokePattern, or focus+click fallback)
• click_coords — mouse click at x, y. Supports button (left/right/middle) and click_count. With relative: true, x/y are offsets from the relative_to origin instead of absolute screen coords (default relative_to: "desktop" = virtual desktop top-left, or "window" = target window top-left).
• set_value / get_value — ValuePattern (edit boxes)
• get_text — TextPattern (read text content)
• focus — move focus to a control
• toggle — TogglePattern (checkbox / switch)
• set_range / get_range — RangeValuePattern (slider / numeric)
• select — SelectionItemPattern (list item)
• scroll — ScrollPattern (amount, scroll_axis)
• expand / collapse — ExpandCollapsePattern

Window actions

• window_minimize / window_maximize / window_restore / window_close
• window_move — move window to (x, y)

find / list_controls report each control's supported patterns.

Architecture

• Reuses pi-web-access as an npm dependency (bundledDependencies), calling its default extension export.
• index.ts wires webAccess(pi) plus registerBrowser, registerScreen, and registerUiAutomation.
• Local modules:
  • browser.ts → registerBrowser (browser)
  • screen.ts → registerScreen (take_screenshot, hide_window)
  • ui-automation.ts → registerUiAutomation (ui_automation)

Install

As a pi package

Add the repo to pi's settings.json:

{
  "extensions": ["D:/pi-automation"],
  "packages": []
}

Copy to the auto-discovery folder

Copy this package to pi's global extensions dir and install deps:

C:\Users\Admin\.pi\agent\extensions\pi-automation\
cd pi-automation && npm install --omit=dev

Avoid tool name collisions

This package registers browser, take_screenshot, hide_window, and the web tools. Disable old sources before enabling it:

• ~/.pi/agent/extensions/browser-cdp.ts (browser)
• ~/.pi/agent/extensions/screen-tools.ts (take_screenshot, hide_window)
• settings.json packages: remove "npm:pi-web-access"

Then run /reload.

Development

cd D:/pi-automation
npm install --omit=dev      # install runtime deps
npm run typecheck           # optional TS check
git add -A && git commit -m "..."   # commit

To test loading in pi: pi -e D:/pi-automation, or /reload.

Notes

• ui_automation, take_screenshot, hide_window are Windows-only; other platforms register no-op/not available tools.
• Reusing pi-web-access also brings its commands and shortcuts.