@intuned/computer-use

Intuned SDK computer-use agent (BETA): drive a desktop with any vision-capable model — screenshots in, mouse/keyboard actions out, executed through xdotool on the X display the process can reach. Built on the AI SDK, routed through the Intuned AI gateway (or a direct provider API key). For driving a Playwright page instead, see @intuned/browser-use.

import { executeTask } from "@intuned/computer-use";

const result = await executeTask({
  task: "Open the browser, search for 'weather in Amman', and report the temperature.",
  maxTurns: 30,
  timeoutInMs: 300_000,
});
console.log(result.finalText, result.totalCostInCents);

Highlights

• Provider-agnostic tools: one function tool per computer action, named like Anthropic's computer toolset (screenshot, left_click, type, key, scroll, ...), so Claude, GPT and Gemini all see the same tools. Actions issued in one turn run in order and stop at the first failure.
• Screenshots on demand: only scroll and wait return a screenshot; the agent calls screenshot after an action that changes the screen.
• Batched actions: a batch tool runs several actions in one call, in order, stopping at the first failure.
• Simple screenshots: ImageMagick's import -window root, scaled with convert, the same approach as Anthropic's computer-use-demo. Displays larger than XGA/WXGA are scaled down for the model and coordinates scaled back up.
• Structured results: pass an outputSchema (JSON schema) and get a schema-validated output object back.
• Extensible: add your own function tools via tools.

Requirements

• xdotool and ImageMagick (import, convert) on the machine. Do not substitute xwd (its multi-visual dump blacks out windows with their own visual, e.g. Wine dialogs) or scrot (its shared-memory grab crashes sandboxed TigerVNC servers).
• An X display: set DISPLAY_NUM, and WIDTH / HEIGHT, or pass the same via the display option; the size falls back to xdotool getdisplaygeometry.

Versions

Versions 0.1.x of this package were the Playwright-page agent, which now lives at @intuned/browser-use. From 0.2.0 on, @intuned/computer-use drives a desktop through xdotool as described here.

Development

yarn install
yarn build   # tsc + babel into dist/
yarn lint