vision-auto

vision-auto is a compact desktop visual automation library for Node.js.

It captures the desktop, matches template images with OpenCV, and exposes simple actions on matched regions such as highlight and click.

Current platform support: Windows only.

What it does

• Desktop template matching with confidence threshold
• Wait-until-visible and wait-until-vanish workflows
• Region actions: highlight, left-click, right-click, and offset-click relative to match center
• Optional center marker for visualizing the exact match and click position
• Promise-friendly API for chaining and explicit control

Install

npm install vision-auto

Quick Start

import { Screen } from 'vision-auto';

const screen = new Screen();

const region = await screen.wait('images/button.png');
await region.highlight({ showCenter: true });
await region.click();

Core API

• new Screen(): creates a screen automation instance.
• screen.find(imagePath, options?): performs a single lookup on the current desktop screenshot. Returns Promise<MatchResult | null>.
• screen.findAll(imagePath, options?): finds all matches above the confidence threshold. Returns Promise<MatchResult[]>.
• screen.wait(imagePath, options?): polls until a match appears or timeout is reached. Returns WaitHandle.
• screen.waitVanish(imagePath, options?): polls until the target image is no longer found. Returns Promise<void>.
• screen.capture(filePath?): captures the desktop as a PNG buffer. Returns Promise<Buffer>.

MatchResult

When matching succeeds, you get a MatchResult object:

• region: { x, y, width, height }
• center: { x, y } (desktop absolute coordinates)
• confidence: similarity score (0 to 1)
• click(button?, offset?): left-click by default, supports 'right', and supports directional pixel offsets from center
• highlight(options?): temporary border highlight with transparent center; set showCenter: true to display a small red cross at the exact center

Offset behavior for click(button?, offset?):

• right: increases x
• left: decreases x
• down: increases y
• up: decreases y

Final click position:

• x = center.x + (offset.right ?? 0) - (offset.left ?? 0)
• y = center.y + (offset.down ?? 0) - (offset.up ?? 0)

Example:

const region = await screen.wait('images/menu-item.png');

console.log(region.center.x, region.center.y);
await region.highlight({
	duration: 1200,
	color: '#FF0000',
	lineWidth: 4,
	showCenter: true,
});
await region.click('right');
await region.click('left', { right: 16, down: 8 });
await region.click('left', { up: 12, left: 10 });

Highlight Center Marker

Set showCenter: true to draw a small red cross at the center of the matched region. This marker identifies the default position used by click().

await region.highlight({ showCenter: true });

WaitHandle click example:

await screen.wait('images/ok-button.png').click('left', { right: 20 });

Manual API Test

The project includes a manual end-to-end script that exercises the latest wait(), highlight(), and click() APIs.

1. Place the template image at button.jpg in the project root.
2. Keep the matching target visible on the desktop.
3. Run:

npm run test:latest-api

The script waits for button.jpg, highlights the match with its center marker, and then performs a real left click at the match center.

Defaults

• screen.wait() defaults: confidence: 0.85, timeout: 10000 ms, interval: 500 ms
• region.highlight() defaults: duration: 1000 ms, color: #FF0000, lineWidth: 4, showCenter: false

Types

The package exports these primary types:

• Screen
• MatchResult
• MatchedRegion
• FindOptions
• WaitOptions
• WaitHandle
• Region
• HighlightOptions
• MouseButton
• ScreenConfig
• TimeoutError

Platform Notes

• The library currently supports Windows only.
• Highlight overlay also depends on Windows PowerShell and the desktop environment.
• Ensure template images and desktop rendering scale are consistent for stable matching.

License

This package is proprietary and closed-source.

• npm artifacts ship compiled output only.
• Source code is not distributed.
• See LICENSE.md for details.

Acknowledgements

This project was inspired by SikuliX. Thanks to the SikuliX project and its contributors for the ideas and groundwork in desktop visual automation.