computer-cli

computer drives desktop applications from the command line — macOS apps through Accessibility, Windows apps through UI Automation over SSH, and any GUI desktop over VNC. It reads the accessibility tree, clicks elements by id or coordinate, types text, sends key chords, takes screenshots, and can run an autonomous task with an embedded model loop. It talks to a local native helper daemon on macOS, an SSH tunnel to the Windows driver, or the RFB protocol for VNC targets.

Install

npm install -g @phnx-labs/computer-cli
computer --version

npm installs a small Node launcher plus one compiled binary for your platform (macOS arm64/x64, Linux arm64/x64, or Windows x64). No TypeScript sources or source maps ship in the package.

First run (macOS)

computer setup                      # download and verify the signed native helper
computer start                      # activate the helper daemon via launchd
computer apps                       # list apps the daemon may drive
computer screenshot -o desktop.png  # capture the display
computer describe                   # dump the accessibility tree of the front app
computer click --id <element>       # click an element from the tree
computer type --id <field> "hello"  # set a field value
computer key "cmd+shift+s"          # send a key chord
computer stop                       # deactivate the daemon

setup downloads the signed native helper to /Applications/. macOS requires the user to grant Accessibility and Screen Recording access; setup cannot do that automatically. start activates the helper daemon via launchd; stop deactivates it. computer status reports installation, connection and trust state.

Remote targets

--host selects a remote desktop using the address grammar shared by every Phoenix CLI:

# Windows: native UI Automation driver over SSH
computer screenshot --host ssh://user@winbox
computer click --host ssh://user@winbox --id <element>

# VNC: any GUI desktop (Linux, macOS, Windows) over RFB
computer screenshot --host vnc://host:5900
computer click --host vnc://host:5900 --x 512 --y 384
computer type-text --host vnc://host:5900 --text "hello"
computer key --host vnc://host:5900 --keys enter

VNC provides screenshot and coordinate input (click --x --y, type-text, key). Accessibility verbs (describe, get-text, click --id, type --id) require a native driver and are not available over VNC.

computer run drives an app from a natural-language task using an embedded model loop over the computer verbs.

Agent tools and isolated Linux desktops

computer mcp serves tools over stdio to your existing agent. It does not make model API calls. Input tools return a compact screenshot as an actual MCP image alongside action metadata. computer run remains the opt-in embedded model loop; it sends the latest screenshot as an image to its configured provider.

On Linux, each new MCP task gets its own desktop by default. Install Xvfb, x11vnc (with Unix-socket support), xauth, and util-linux (flock, setpriv). These displays start empty; launch an application with launch --path.

computer mcp                           # create an isolated Linux task
computer task start                    # create a task for CLI use; prints its ID
computer launch --path /path/to/app --desktop-task TASK_ID
computer click --x 400 --y 300 --screenshot --desktop-task TASK_ID
computer mcp --desktop-task TASK_ID     # reconnect another agent to that task
computer task list
computer task stop TASK_ID

Save the default in ~/.agents/.computer/config.json, or override it when creating a task. An override never changes the saved setting or an existing task's mode.

computer config desktopMode shared
computer task start --desktop-mode isolated
computer mcp --desktop-mode shared --host vnc://localhost:5901

One private local service coordinates Linux tasks. Clients sharing a task hold its desktop lock through target selection, input and capture. Separate isolated desktops run independently; shared-mode tasks are serialized. Direct commands without --desktop-task do not participate in that coordination. Disconnecting one client does not stop another client's task. Idle tasks expire after 15 minutes.

Isolation covers the X display, windows, focus and pointer, not the host user, filesystem, application profiles or credentials. Applications that reuse a global profile may require their own separate-profile launch configuration. Private task handles prevent accidental cross-task access; they are not a security boundary against hostile code running as the same OS user. No TCP port is opened for an isolated desktop. Native macOS shared-mode MCP uses the existing helper; managed task provisioning currently requires Linux.

Action observations and local OCR

computer click --host vnc://localhost:5901 --x 400 --y 300 --screenshot result.png --max-dimension 1280 --json
computer screenshot --host vnc://localhost:5901 --ocr --out screen.png --json

--ocr uses local Tesseract and includes text, word bounds and confidence, without a model call. For input commands it implies a post-action capture. OCR coordinates refer to the original desktop; screenshot scale and scale_y map that desktop into the resized image. Images preserve aspect ratio and default to a maximum edge of 1280 pixels for action/MCP observations. A screenshot is an observation, not a guarantee that an application's asynchronous work has finished.

If capture fails after a successful action, the action remains successful and the capture error is returned separately. Retry observation, not the action. Native compact capture requires a helper supporting max_dimension; an older helper that ignores the requested bound reports helper_update_required.

How fast are action observations?

In a September 15, 2026 Linux arm64 benchmark, the installed 0.1.5 CLI was compared with the 0.1.6 candidate at 60ca877. The candidate was not released or installed at measurement time. Gedit displayed the real 729-line src/commands/actions.ts; each sample sent Page Down and captured the result on the same owned, isolated 1600 × 1000 desktop.

| Workflow | Median | Observed range | Samples | | --- | ---: | ---: | ---: | | 0.1.5: separate action and screenshot | 4,494 ms | 4,453–4,552 ms | 8 | | Candidate: separate action and screenshot | 515 ms | 454–691 ms | 8 | | Candidate: combined action and screenshot | 712 ms | 693–947 ms | 8 | | Candidate: MCP image response | 533 ms | 496–578 ms | 8 | | Candidate: MCP image + local OCR | 1,205 ms | 1,111–1,364 ms | 8 |

The candidate MCP workflow had 88.1% lower median latency (8.4× speedup) than the released separate-call workflow. Most of that improvement also exists in the candidate's separate CLI calls, following the connection-close timer fix; it is not an MCP-only speedup. Combined capture was 38% slower than candidate separate calls here, and MCP was not demonstrably faster than candidate separate calls. OCR added 672 ms to the MCP median.

Separate screenshots were full-size; combined/MCP observations used the default 1280 × 800 image and a 150 ms post-input receive window. MCP PNGs were 21.3% smaller (93,406 versus 118,695 bytes), excluding protocol/base64 overhead. These compare default workflows, not identical capture settings.

Each mode had one warmup; measured order rotated between rounds. Timings include CLI process start/exit or a tool request on an already-connected MCP client. Desktop/app startup, the initial 67 ms MCP connection and harness verification are excluded. All 40 captures matched the expected source section using case-insensitive OCR anchors; original case-sensitive misses were retained in the raw results. The busy host and eight samples per mode make this a directional comparison, not a production tail-latency estimate. No model calls, token savings, concurrent-task throughput, macOS latency or Windows latency were measured.

See the benchmark evidence on PR #14 for the recorded run and delivery limits. Source links require repository access.

Command reference

scripts/generate-cli-docs.sh                              # refresh docs/
scripts/generate-cli-docs.sh --check                      # fail on stale docs
scripts/generate-cli-docs.sh --out-dir .agents/scratch/cli-docs  # preview

Run after bun install --frozen-lockfile. The script works from any directory; --out-dir is relative to your current directory. Open command-reference.html from the output folder to browse the tree and search commands. CI uploads a computer-cli-docs artifact on pull requests and pushes to main or release/**. It contains the HTML reference plus Markdown and JSON indexes. Tests and releases check the committed files; commit regenerated docs/ with command changes. Release checks also run with --skip-tests / --skip-build.

docs/command-reference.html lists every command with its arguments and options, searchable, with the command tree in a sidebar. docs/command-index.md is the one-line-per-command index, and docs/command-index.json the structured form. scripts/generate-cli-docs.sh regenerates all three from the registered command tree; scripts/test.sh fails when they are stale.

Using it through agents-cli

Agents CLI ships agents computer as a thin consumer of this binary. It forwards every verb unchanged and adds fleet device discovery (--device becomes a resolved --host), agent-session identity, and cross-machine session history. The seam between the two is documented in docs/integration.md. Transports, addresses and verbs are explained in docs/computer.md.

Platform support

| Platform | CLI architectures | Desktop driver | | --- | --- | --- | | macOS | arm64, x64 | Native Accessibility helper (local) | | Windows | x64 | UI Automation driver over SSH | | Linux | arm64, x64 | VNC; isolated task desktops via Xvfb (no native Accessibility) |

These badges describe platform support, not feature parity. Managed isolated task provisioning is Linux-only. The measurements above cover Linux/VNC, not the native macOS or Windows paths.

Do macOS and Windows need native helpers?

Yes, for native element-level automation, and both helpers already live in this repository, extracted from Agents CLI. Installing Agents CLI is not required.

• macOS: the Swift helper supplies Accessibility, ScreenCaptureKit capture and CoreGraphics input. Its signed app bundle keeps a stable identity for the user's Accessibility and Screen Recording grants. The CLI downloads/verifies it during setup and manages it through launchd.
• Windows: the C#/.NET helper supplies UI Automation, screen capture and SendInput. Setup provisions a scheduled task in the logged-in interactive desktop session. The CLI reaches its loopback service over an authenticated SSH tunnel; an SSH login alone is not a GUI session.
• Linux/VNC: no macOS or Windows helper is needed. A VNC server supplies screenshot and coordinate input; isolated Linux tasks additionally need the dependencies listed above.

Native helpers retain their separate, immutable computer-mac/v* and computer-win/v* release channels. An ordinary CLI release does not rebuild them.

Troubleshooting

• Accessibility not trusted (macOS). computer status shows whether the helper has Accessibility and Screen Recording access. Grant both in System Settings > Privacy & Security. The helper binary path must match the grant; computer setup after a helper upgrade may require re-granting.
• VNC password. Pass --vnc-password <pw> or set COMPUTER_HELPER_VNC_PASSWORD. A VNC server with no authentication needs neither.
• unsupported_over_vnc. Accessibility verbs (describe, get-text, click --id, type --id) require a native driver. Over VNC, use coordinate input (click --x --y, type-text --text, key --keys).
• Windows SSH tunnel. computer start --host ssh://user@winbox opens the tunnel. The Windows driver runs in the interactive desktop session, so the SSH user must have an active logon session.

Development

bash scripts/install.sh   # install dependencies
bash scripts/test.sh      # typecheck + tests
bash scripts/build.sh     # compile this platform's binary into dist/

scripts/build.sh bundles and minifies src/, obfuscates the bundle offline with a pinned tool and fixed seed, then compiles a bytecode binary per target. Run dist/<platform>-<arch>/computer directly during development.

Release with the repository's own script:

bash scripts/release.sh                 # print the release plan (dry run)
secrets exec npmjs.com -- bash scripts/release.sh --confirm

Release requires a clean linked worktree at the reviewed origin/main commit. It builds every platform, verifies packed file lists and immutable package integrity, publishes the launcher and per-platform packages, and checks a registry install. Live desktop verification remains part of delivery.

Minification, obfuscation and bytecode compilation raise the bar for casual inspection; they are not a guarantee against reverse engineering. Keep credentials out of the binary.

License

Functional Source License 1.1 with an Apache 2.0 future grant (FSL-1.1-Apache-2.0).

Browse run history

computer sessions opens a searchable picker: arrows move, Enter shows run details and saved screenshots, and Escape goes back. --search <text> also filters scripted output, --json remains a row array, and --no-interactive prints the table. --since <when> and --until <when> keep runs whose last activity falls in that window and accept the same values as sessions: 2h, 7d, 4w, 1mo, 1y or an ISO date. An unreadable value exits 2. --limit <n> caps the table (default 50) and, when given, --json; when rows are left out, the count and the --limit that shows them go to stderr, so piped stdout carries rows only. The picker header lists the active filters. --open latest (or a screenshot filename) opens a recorded capture. Live Linux desktops remain under computer task start/list/stop.

Computer owns 365-day run summaries under .history/computer/history.db. Counts survive the 30-day action ledger; summaries retain the latest 120 actions and 120 screenshot locations per run. Typed text is never stored. Existing ledger entries and legacy Agents summaries are adopted without modifying their source. Legacy runs older than 365 days are skipped. The automatic import runs once; if the legacy database cannot be read it is reported once, and computer sessions migrate [--from /path/to/sessions.db] retries. The command safely repeats the summary import. Sessions need not be installed.