OpenCode Windows UI Automation

Framework-independent, human-like Windows UI automation and visual frontend testing for OpenCode.

The extension does not replace OpenCode's terminal, compiler, or build workflow. It gives the agent the missing ability to interact with and verify visible Windows frontends: web applications, VS Code extension development hosts, Electron apps, Win32, WPF, WinUI, Qt, native C/C++ applications, and other desktop windows.

The plugin supports both:

• OpenCode running natively on Windows.
• OpenCode running inside WSL while controlling the real Windows desktop through WSL/Win32 interoperability.

System requirements

Runtime requirements

• Windows 10 version 1703 (build 15063) or newer, or Windows 11.
• OpenCode 1.18.0 or newer.
• An interactive Windows desktop session. Headless Windows Server Core sessions are not supported.
• x64 is the primary tested architecture. Windows ARM64 is expected to work but should be treated as experimental until verified on real ARM64 hardware.
• A model with image input for direct screenshot vision, or any text model using the included offline OCR and Windows Accessibility tools.

Additional requirements when OpenCode runs in WSL

• WSL 2 is recommended.
• For a new WSL installation, Microsoft requires Windows 10 version 2004/build 19041 or newer, or Windows 11, for wsl --install.
• Standard Windows interoperability must be enabled so WSL can execute powershell.exe and cmd.exe.
• wslpath must be available. It is included with normal WSL installations.
• OpenCode may work in a project under /mnt/c/... or directly in the Linux filesystem such as ~/code/....
• Native Linux outside WSL is not supported. The package can be installed on Linux so that WSL works, but runtime Windows tools reject a normal Linux environment.

Development/build requirements

These are required only when building or testing this repository from source:

• Node.js 20 or newer.
• npm.
• For npm run test:visible: native Windows or WSL with Windows interoperability and an interactive Windows desktop.

How WSL support works

When OpenCode runs in WSL, the plugin itself runs in the Linux OpenCode process, but Windows UI operations are delegated to Windows PowerShell through WSL interoperability.

• Mouse, keyboard, clipboard, Windows UI Automation, screen capture and pixel reads run in Windows PowerShell and Win32.
• OCR and PNG comparison continue to run in the OpenCode/WSL process.
• Linux file paths are converted to Windows paths only when a Windows process needs to access them.
• No Windows user name, home directory, or C:\Users\... path is hardcoded.

Install in native Windows OpenCode

Open PowerShell:

opencode --version
opencode plugin -g [email protected]
opencode

To replace an already configured version:

opencode plugin -g -f [email protected]

OpenCode writes the plugin to the global config and installs npm plugin dependencies automatically.

Install in OpenCode running in WSL

Open the WSL terminal:

opencode --version
printf 'WSL_DISTRO_NAME=%s\n' "$WSL_DISTRO_NAME"
powershell.exe -NoLogo -NoProfile -Command '$PSVersionTable.PSVersion.ToString()'
cmd.exe /c ver
wslpath -w "$PWD"
opencode plugin -g [email protected]
opencode

If the package is already configured:

opencode plugin -g -f [email protected]

Smoke test from OpenCode

After a provider is configured, this command starts a non-interactive OpenCode run and asks the agent to exercise the Windows tools:

Use win_screen_info, then win_windows, then take a screenshot named opencode-plugin-smoke.png with win_screenshot. Report whether all three calls succeeded and return the screenshot path.

Run it from PowerShell or WSL with:

opencode run "Use win_screen_info, then win_windows, then take a screenshot named opencode-plugin-smoke.png with win_screenshot. Report whether all three calls succeeded and return the screenshot path."

Tools

| Tool | Purpose | |---|---| | win_screen_info | Displays, working areas, and cursor position | | win_screenshot | Full-desktop or regional PNG screenshot returned as an OpenCode image attachment | | win_mouse | Move, click, double-click, mouse-down, mouse-up, smooth drag-and-drop, and scroll | | win_keyboard | Unicode text entry and keyboard shortcuts | | win_clipboard | Read and write Unicode clipboard text | | win_windows | List visible application windows | | win_window_action | Focus, minimize, maximize, and restore windows | | win_accessibility_tree | Inspect up to 500 Windows UI Automation elements and their screen bounds | | win_accessibility_action | Focus, invoke, toggle, or set the value of accessible UI elements | | win_wait_for_ui | Wait for a window or accessible element | | win_pixel_color | Read a screen pixel as RGBA and hexadecimal color | | win_ocr | Offline English/German OCR with word confidence and bounding boxes | | win_visual_compare | Pixel-level PNG regression comparison with optional diff image |

All tool names are prefixed with win_ to avoid collisions with OpenCode's built-in tools and other plugins.

How an agent tests a frontend

1. Build and start the application with OpenCode's built-in terminal tool.
2. Wait for its window with win_wait_for_ui.
3. Inspect standard controls with win_accessibility_tree or use win_ocr for custom-drawn interfaces.
4. Interact using accessibility actions or human-like mouse, keyboard, clipboard, scroll, and drag-and-drop operations.
5. Capture screenshots and verify pixels, OCR text, accessible state, or visual differences.

The coordinate tools operate at the Windows desktop level and are therefore independent of the frontend framework.

Build and verification from source

From the repository directory:

npm ci
npm run build
npm test
npm run test:visible
npm pack

npm test always checks the plugin/tool surface and, on native Windows or WSL, also runs safe Windows screenshot/visual checks. npm run test:visible opens a temporary native Windows frontend, moves the mouse, changes the clipboard, tests the UI tools, restores the original mouse/clipboard state, and removes its temporary files.

See PUBLISHING.md for the exact native-Windows, WSL, clean-instance, build and publishing command sequences.

Publishing

The npm package name is opencode-windows-ui-automation.

npm login
npm publish --access public

The package declares win32 and linux as installable npm platforms because WSL reports itself as Linux. Runtime execution still requires either native Windows or a WSL environment connected to a Windows host.

Maintainer

Joel Buchholz

License

MIT © 2026 Joel Buchholz