pi-codex-computer-use

A Pi extension that exposes OpenAI Codex Computer Use as Pi-native tools, so Pi agents can inspect and operate local macOS apps — reading accessibility trees and screenshots, then clicking, typing, scrolling, and pressing keys.

The extension talks to Codex app-server, which owns the Computer Use plugin lifecycle, MCP startup, and native permission flow:

Pi tool -> Codex app-server -> computer-use MCP server -> Codex Computer Use macOS app/service -> local desktop

See docs/ARCHITECTURE.md for how the pieces fit together.

Requirements

• macOS
• Pi
• Codex CLI on PATH as codex
• Codex.app installed, preferably at /Applications/Codex.app
• Codex Computer Use plugin installed, or installable from the bundled marketplace (/computer-use install handles this)
• macOS Accessibility and Screen Recording permissions granted to Codex Computer Use when prompted
• Preferably a vision-capable model, because computer_use_get_app_state returns screenshots

Installation

Install from npm:

pi install pi-codex-computer-use

Or install a local checkout:

pi install ./path/to/pi-codex-computer-use

Or add it to project settings (.pi/settings.json):

{
  "packages": ["pi-codex-computer-use"]
}

Then, inside Pi, verify the backend is ready:

/computer-use status

If the plugin is missing, /computer-use install adds the bundled Codex marketplace, installs the computer-use plugin, and reloads MCP servers.

Usage

Ask Pi to do something on your desktop:

Use Computer Use to open Calculator and compute 4 + 5.

The bundled computer-use skill teaches the model the standard loop: get app state, act, get app state again, verify.

Slash commands

| Command | What it does | | --- | --- | | /computer-use status | Read-only check of Codex CLI, app-server, plugin, MCP server, and tools | | /computer-use install | Add bundled marketplace if needed, install/enable the plugin, reload MCP servers | | /computer-use reload | Reload Codex MCP servers and recheck status | | /computer-use restart | Shut down the runtime; next tool use lazy-starts it again | | /computer-use enable | Re-add the Computer Use tools to Pi's active tool set | | /computer-use disable | Remove the tools and shut down the runtime (session-local) | | /computer-use diagnose | Status plus environment and permission hints |

Lifecycle

The extension lazy-starts Codex app-server only when a command or computer_use_* tool needs it. The Pi footer reflects the current state:

• Computer Use: idle — nothing running (normal startup state)
• Computer Use: <app> — a tool call is operating that app
• Computer Use: permission… — waiting on a user permission prompt
• Computer Use: ready — backend up, last call succeeded
• Computer Use: disabled — tools removed via /computer-use disable

The runtime shuts itself down when the agent finishes its turn, and after an idle timeout (default 10 minutes) as a fallback.

Tools

All tools proxy one-to-one to the Codex computer-use MCP server. Calls are serialized through a single queue — desktop actions never run concurrently, even if the model emits parallel tool calls.

| Tool | Purpose | | --- | --- | | computer_use_list_apps | List running/launchable apps | | computer_use_get_app_state | Accessibility tree + screenshot for an app | | computer_use_click | Click an element index or coordinates | | computer_use_type_text | Type into the focused field | | computer_use_press_key | Press a key or key combo (e.g. super+s) | | computer_use_scroll | Scroll a scrollable element | | computer_use_drag | Drag between points | | computer_use_set_value | Set a control's value directly | | computer_use_select_text | Select text in a text area | | computer_use_perform_secondary_action | Element's secondary (e.g. context-menu) action |

Configuration

| Environment variable | Default | Effect | | --- | --- | --- | | PI_CUA_DEBUG | unset | 1 writes debug logs to stderr | | PI_CUA_LOG | unset | Path to append structured debug logs | | PI_CUA_IDLE_TIMEOUT_MS | 600000 | Idle time before the runtime auto-shuts-down | | PI_CUA_DEV_AUTO_ACCEPT_APPS | unset | Dev-only comma-separated app allowlist for auto-accepting app access prompts |

Safety

Computer Use operates your live desktop. Defaults are conservative:

• Codex app-server permission requests ("Allow Codex to use Finder?") are bridged to Pi UI confirmations.
• With no UI available (e.g. print mode), permission requests fail closed and are declined.
• The bundled skill instructs the model to confirm with you before risky UI actions: sending messages, submitting forms, deleting data, payments, account/security changes, installing software, or transmitting sensitive data.
• Logs redact screenshot/base64/text-like fields by default.

PI_CUA_DEV_AUTO_ACCEPT_APPS is for development against safe apps (Finder, TextEdit, Calculator) only. Note that accepted app access can persist in Codex's own permission store across runs. Do not use broad auto-approval.

Troubleshooting

Run /computer-use diagnose first — it prints status plus actionable hints.

| Symptom | Likely cause / fix | | --- | --- | | codex_missing | Codex CLI not on PATH; install Codex CLI | | marketplace_missing | Codex.app not at /Applications/Codex.app, or bundled marketplace path missing | | plugin_not_installed / plugin_disabled | Run /computer-use install | | mcp_missing | Plugin installed but MCP server not running; try /computer-use reload | | Tool calls hang or error with permission messages | Grant Accessibility and Screen Recording to Codex Computer Use in macOS System Settings | | Permission prompts never appear in print mode | Expected: no-UI mode fails closed; run interactively or use the dev allowlist for safe apps | | thread not found errors | Handled automatically (thread reset + one retry); persistent failures suggest app-server instability — /computer-use restart |

Debug logging:

PI_CUA_DEBUG=1 PI_CUA_LOG=/tmp/pi-codex-computer-use.log pi -e .

Limitations

• macOS only.
• Requires the Codex app and its bundled Computer Use plugin; this extension does not implement its own desktop automation.
• Codex app-server and Computer Use APIs may drift between Codex versions; /computer-use status reports versions and tool lists to help spot drift.
• /computer-use disable is session-local; new Pi sessions re-enable the tools.

Contributing

See CONTRIBUTING.md for the development workflow and docs/TESTING.md for the test matrix. Contributions from humans and automated agents are welcome — agents should also read AGENTS.md.

License

MIT