augui

Fast GUI automation for Node.js on Windows — mouse, keyboard, and image matching with optional GPU acceleration. Built on rustautogui via napi-rs, so there's no Python, no OpenCV, and nothing to compile at install time.

import { agui } from 'augui';

await agui.click('assets/save-button.png', { timeout: 5 });
await agui.type('report.txt', 'assets/filename-field.png');
agui.press('enter');
await agui.waitGone('assets/spinner.png', 30);

Install

npm install augui

Windows x64 only. The compiled binary ships with the package — you don't need Rust.

Quick start

Automation works by finding a picture of something on screen, then acting on where it was found. So the first step is making that picture:

import { agui } from 'augui';

// Cut a piece of your screen out and save it as a template.
await agui.captureTemplate('assets/button.png', {
  x: 450, y: 220, width: 120, height: 40,
});

// Later — find it and click it.
const hit = await agui.find('assets/button.png');
if (hit) await agui.click(hit);

Pick a patch with visual detail. A flat area of one colour has no unique signal and won't be found.

Match coordinates are the centre

find returns the middle of what it matched, so the result goes straight into a click with no arithmetic:

const hit = await agui.find('assets/button.png');
// hit -> { x: 510, y: 240, confidence: 0.98 }
await agui.click(hit);

Finding things

await agui.find('img.png');                    // Match | null
await agui.find('img.png', { timeout: 10 });   // retry for 10s
await agui.findOrThrow('img.png');             // throws NotFoundError
await agui.findAll('img.png');                 // Match[], best first
await agui.exists('img.png');                  // boolean
await agui.waitFor('img.png', 30);             // appear, or throw
await agui.waitGone('img.png', 30);            // disappear, or throw

find and exists are the forgiving ones — they answer null/false when something isn't there. findOrThrow, waitFor, click and moveTo throw NotFoundError, which suits a script that should stop when the UI isn't where you expect.

import { NotFoundError } from 'augui';

try {
  await agui.click('assets/submit.png', { timeout: 5 });
} catch (e) {
  if (e instanceof NotFoundError) console.log('submit never appeared');
  else throw e;
}

Options

Every find-style call takes the same options:

await agui.find('img.png', {
  region: { x: 0, y: 0, width: 800, height: 600 },  // search here only
  matchMode: agui.fastestMode,                      // which algorithm
  precision: 0.8,                                   // how hard to look
  minConfidence: 0.95,                              // how good a match to keep
  timeout: 10,                                      // keep retrying
});

precision and minConfidence are different knobs. precision scales an internal threshold and means different things in different match modes — Segmented is noticeably stricter than Fft. minConfidence is a real floor on the reported score and means the same thing everywhere. Reach for minConfidence when you care about match quality.

Clicking and moving

click is really a move-and-click. It finds the target, moves the cursor there, and clicks - all in one call. There's no need to moveTo first, and doing so searches for the image twice:

// what you want
await agui.click('assets/save.png');

// same thing, but searches twice and takes ~2x as long
await agui.moveTo('assets/save.png');
await agui.click('assets/save.png');

Use moveTo only when you want to hover without clicking.

await agui.click('assets/save.png');                  // find, then click
await agui.click({ x: 500, y: 300 });                 // click a point
await agui.click(hit);                                // click a funded Match
await agui.click('img.png', { timeout: 5 });          // wait for it first
await agui.click('img.png', { button: 'right' });
await agui.click('img.png', { moveTime: 0.4 });       // glide, then click

await agui.doubleClickOn('assets/icon.png');
await agui.rightClickOn('assets/file.png');

await agui.moveTo('assets/menu.png');                 // point, don't click
await agui.moveTo({ x: 800, y: 400 }, { moveTime: 0.5 });

await agui.drag('assets/file.png', { x: 1200, y: 600 });

Typing

await agui.type('hello world');                       // type at the cursor
await agui.type('hello', 'assets/search-box.png');    // click there, then type
await agui.replaceText('new value', 'assets/field.png'); // select all, then type

agui.press('enter');
agui.press('tab', 'tab', 'enter');                    // several, in order
agui.hotkey('ctrl', 'c');
agui.hotkey('ctrl', 'shift', 'esc');

agui.copy();
agui.paste();
agui.selectAll();

agui.keyDown('shift');                                // hold
agui.keyUp('shift');

Key names are strings: 'enter', 'tab', 'esc', 'f5', 'up', 'ctrl', 'shift', 'alt', 'win', 'a'…'z', '0'…'9'. An unknown name throws.

Screen

agui.screen;                            // { width, height }
agui.mouse;                             // { x, y }

await agui.screenshot('shot.png');
await agui.captureTemplate('assets/btn.png', { x: 600, y: 300, width: 200, height: 80 });
await agui.cropImage('shot.png', 'btn.png', { x: 600, y: 300, width: 200, height: 80 });

agui.scrollDown(3);
agui.scrollUp(3);
agui.scrollLeft(1);
agui.scrollRight(1);

Speed

Search a region, not the whole screen

This is the biggest lever there is — cost scales with the area you scan. On a 3440×1440 screen:

| search area | time | |---|---| | full screen | 711ms | | 1000×600 region | 76ms — 9x faster | | 400×200 region | 40ms — 18x faster |

await agui.find('assets/button.png', {
  region: { x: 0, y: 0, width: 1000, height: 600 },
});

Match modes

The default is Segmented (CPU). It does not auto-pick the fastest. Full-screen searches on a 3440×1440 display with an RTX 5070:

| mode | 200×80 | 600×400 | cpu | |---|---|---|---| | Segmented (default) | 391ms | 2883ms | ~21–27 cores | | Fft | 696ms | 708ms | ~1 core | | SegmentedOcl (GPU) | 86ms | 137ms | ~0.9 core |

import { agui, MatchMode } from 'augui';

await agui.find('btn.png');                                        // Segmented
await agui.find('btn.png', { matchMode: MatchMode.Fft });
await agui.find('btn.png', { matchMode: MatchMode.SegmentedOcl }); // GPU
await agui.find('btn.png', { matchMode: agui.fastestMode });       // portable

agui.fastestMode is SegmentedOcl on a GPU-enabled build and Fft otherwise, so the same line works everywhere. Set it once:

import { agui, type FindOpts } from 'augui';

const FAST: FindOpts = { matchMode: agui.fastestMode };

await agui.click('assets/save.png', { ...FAST, timeout: 5 });

Rules of thumb:

• Large templates — Fft. Segmented slows down ~4x; FFT barely notices.
• Tight CPU budget — Fft uses one core, Segmented uses all of them.
• Many searches in a loop — a region beats any mode choice.

The default is the safe one rather than the fast one, because GPU modes throw on a CPU-only build. One thing is automatic: if Segmented refuses a template as too detailed, find retries with Fft.

Don't block the event loop

Instant calls are synchronous. Anything with a duration returns a promise and must be awaited:

agui.moveMouseTo(500, 300);                   // instant, sync
agui.leftClick();                             // instant, sync

await agui.moveMouseToAsync(500, 300, 0.5);   // timed — always async
await agui.dragMouseToAsync(800, 600, 0.5);
await agui.sleep(1.5);                        // pause without blocking

The synchronous form with a duration (agui.moveMouseTo(x, y, 0.5)) blocks your whole process for that long. Fine once; not in a loop.

GPU support

GPU matching needs a build with the opencl feature. Check at runtime:

if (agui.openclAvailable()) {
  await agui.printOpenclDevices();   // prints to stdout
  await agui.setOpenclDevice(0);     // pick a device
}

Using MatchMode.SegmentedOcl on a CPU-only build throws with a message explaining why. agui.fastestMode avoids the problem entirely.

Full example

import { agui, NotFoundError } from 'augui';

async function openReport() {
  await agui.click('assets/file-menu.png', { timeout: 5 });
  await agui.click('assets/open.png');
  await agui.type('report.txt', 'assets/filename-field.png');
  agui.press('enter');

  await agui.waitGone('assets/loading.png', 30);

  if (await agui.exists('assets/error-dialog.png')) {
    await agui.click('assets/ok-button.png');
    throw new Error('the file failed to open');
  }
}

openReport().catch((e) => {
  if (e instanceof NotFoundError) console.error('UI was not where expected:', e.message);
  else console.error(e);
  process.exit(1);
});

API reference

Finding

| | | |---|---| | find(image, opts?) | Promise<Match \| null> | | findOrThrow(image, opts?) | Promise<Match> — throws NotFoundError | | findAll(image, opts?) | Promise<Match[]>, best first | | exists(image, opts?) | Promise<boolean> | | waitFor(image, timeout?, opts?) | Promise<Match> — throws on timeout | | waitGone(image, timeout?, opts?) | Promise<void> — throws if still there |

Pointing

| | | |---|---| | click(target, opts?) | Promise<Point> | | doubleClickOn(target, opts?) | Promise<Point> | | rightClickOn(target, opts?) | Promise<Point> | | moveTo(target, opts?) | Promise<Point> | | drag(from, to, opts?) | Promise<void> | | leftClick() rightClick() middleClick() doubleClick() | void | | moveMouseTo(x, y, time?) | void — sync, blocks if time given | | moveMouseToAsync(x, y, time?) | Promise<void> | | moveMouse(dx, dy, time?) / moveMouseAsync | relative | | dragMouseTo(x, y, time?) / dragMouseToAsync | | | scrollUp/Down/Left/Right(n?) | void |

Typing

| | | |---|---| | type(text, into?, opts?) | Promise<void> | | replaceText(text, into?, opts?) | Promise<void> | | press(...keys) | void | | hotkey(k1, k2, k3?) | void | | copy() paste() selectAll() | void | | typeText(text) | void — raw, no click | | keyDown(key) keyUp(key) | void |

Screen

| | | |---|---| | screen | { width, height } | | mouse | { x, y } | | fastestMode | best MatchMode this build supports | | screenshot(path?) | Promise<string> | | captureTemplate(path, region, from?) | Promise<string> | | cropImage(src, dest, region) | Promise<void> | | sleep(seconds) | Promise<void> | | setVerbose(on) | void — let the crate's own logging through | | openclAvailable() | boolean | | printOpenclDevices() | Promise<void> | | setOpenclDevice(i) | Promise<void> |

Types

type Target = string | { x: number; y: number };

interface Match { x: number; y: number; confidence: number }
interface Region { x: number; y: number; width: number; height: number }
interface Point { x: number; y: number }

interface FindOpts {
  region?: Region;
  matchMode?: MatchMode;
  precision?: number;      // default 0.8
  minConfidence?: number;  // default 0
  timeout?: number;        // seconds, default 0
}

interface ClickOpts extends FindOpts {
  button?: 'left' | 'right' | 'middle';
  double?: boolean;
  moveTime?: number;       // seconds; glide instead of jumping
}

Multiple instances

agui is a shared instance and is what you normally want. For a separate one:

import { Agui } from 'augui';
const other = new Agui();

Notes

• Windows x64 only. rustautogui supports macOS and Linux/X11, but this package is built and tested only for Windows.
• Coordinates are physical pixels. On a scaled display, mark the Node process DPI-aware or your coordinates will be off.
• Templates are cached by path, size and modified time, so overwriting a template file is picked up automatically.
• rustautogui's console output is muted. The upstream crate prints Running <mode> mode on every search with no flag of its own to stop it, so this package silences it. agui.setVerbose(true) lets it through if you're debugging a match.

License

MIT