Element Selector SDK

Enterprise-grade UI Automation SDK for Windows with imperative API and full TypeScript support.

Features

• ✅ Imperative API - Full control flow support (if/else, while, try/catch)
• ✅ Type Safety - Complete TypeScript definitions
• ✅ Element as First-Class - Reusable element references
• ✅ Flexible Error Handling - Try/catch instead of auto-exit
• ✅ Humanized Automation - Bezier curves, random delays
• ✅ XPath Support - Powerful element querying
• ✅ Image Matching - Template-based image finding with DPI auto-adaptation
• ✅ Unified Entry - findElement() with :all / :onlyone markers
• ✅ Image Acceleration - accel option for faster repeated element finding

Installation

npm install @element-selector-sdk-nodejs

Quick Start

import { SDK } from '@element-selector-sdk-nodejs';

async function main() {
  const sdk = new SDK({ baseUrl: 'http://localhost:8080' });
  const flow = sdk.flow();
  
  // Activate window
  await flow.window({ title: 'Notepad' });
  
  // Find and click button
  const button = await flow.find('//Button[@Name="Save"]');
  if (await button.isEnabled()) {
    await button.click();
  }
  
  // Type text
  const input = await flow.find('//Edit');
  await input.type('Hello, World!', { humanize: true });
}

main().catch(console.error);

Image Finding

// Find image in current window (requires flow.window() first)
const matches = await flow.findImage('./images/btn.png');
console.log(`Found at (${matches[0].x}, ${matches[0].y})`);

// Click image
await flow.clickImage('./images/btn.png', { clickArea: { left: '20%', right: '20%' } });

// Scroll to find image
await flow.scrollToImage('./images/scroll-target.png', { scrollContainer: '//Document' });

Image Acceleration (accel)

Use accel to cache element images for faster repeated finding:

// First call: UIA find + capture element image
// Subsequent calls: findImage (much faster)
await flow.click('//Button', { accel: true });
await flow.waitFor('//Text', { accel: true, timeout: 5000 });
await flow.exists('//Button', { accel: true });
await flow.scrollToVisible(xpath, null, { accel: true });

How it works:

| 调用 | 模板文件 | 行为 | |---|---|---| | 首次（模板不存在） | 无 | UIA 查找 → 截图缓存 → 返回 | | 后续（模板已缓存） | 有 | findImage 匹配 → 命中返回 / 未命中抛错 | | 图像路径 miss | — | 直接抛错，不回退 UIA（findImage 比 UIA 更快） |

Debug Visualization

click 支持两种独立的高亮可视化，可单独或同时启用：

// flash：点击前闪烁元素矩形框（红色边框，默认 1000ms）
await el.click({ flash: true });

// showDot：点击位置画圆点动画（Direct2D 抗锯齿双环聚焦收缩）
await el.click({ showDot: true });

// 两者同时：先闪框 → 点击 → 画点
await el.click({ flash: true, showDot: true });

// 自定义 flash 持续时间
await el.click({ flash: { timeout: 2000 } });

scrollToVisible 的 scrollEndDetection.saveDebugFrames=true 会保存最近 consecutiveFrames 帧截图到临时目录，标注采样区（红框）/ 被 mask 剔除像素（暗灰）/ 模板命中（绿框），便于调优滚动到底检测算法。

Architecture

findElement(xpath)          ← 标记路由：:all / :onlyone / :first
  → findElementAll          纯 UIA
  → findOne / findFirst     路由层

findOne / findFirst         ← 路由层：按 accel 分派
  → findElementOne          纯 UIA（多个报错）
  → findElementFirst        纯 UIA（多返回第一个）
  → findImageOne            纯图像（不回退 UIA）
  → findElementAndCache     UIA + 自动截图缓存

findElementAll              纯 UIA
findImageOne                纯图像

Unified findElement Entry

// Default: findFirst
await flow.findElement('//Button');

// :onlyone = findOne (error if multiple)
await flow.findElement('//Button:onlyone');

// :all = findAll
await flow.findElement('//Button:all');

Element Methods

| Primary | Alias | Description | |---------|-------|-------------| | text() | getText() | 元素文本 | | bounds() | getRect(), boundingBox() | 位置和尺寸 | | attr(name) | getAttribute(name) | 元素属性 | | parent() | parentElement() | 父元素 | | next() | nextSiblingElement() | 下一个兄弟元素 | | prev() | previousSiblingElement() | 上一个兄弟元素 | | dblclick() | doubleClick() | 双击 | | find(xpath) | locator(xpath) | 子元素查找 | | findAll(xpath) | — | 所有子元素 | | scrollIntoView() | — | 滚动到元素可见（支持 autoDelta） |

scrollIntoView

将屏幕外元素滚动到可视区域：

const el = await flow.find('//Button[@Name="底部按钮"]');
await el.scrollIntoView({ delta: -120, times: 10 });

| 选项 | 类型 | 默认值 | 说明 | |------|------|--------|------| | delta | number | -120 | 滚动量（负=向下，正=向上） | | times | number | 10 | 最大滚动次数 | | autoDelta | boolean | true | 根据容器高度自动计算滚动量 | | deltaFactor | number | 0.8 | autoDelta 乘数因子 | | timeout | number | 10000 | 超时毫秒 | | propNames | string[] | — | 用于生成唯一 wait XPath 的属性名 |

Documentation

• 📖 Migration Guide - Detailed migration instructions from v1.x
• 📖 Implementation Summary - Technical implementation details
• 📖 Examples - Complete working examples

Examples

Check out the examples directory for complete working examples:

# Quick start
npm run example:quick

# Advanced usage
npm run example:advanced

# Full demo
npm run example:full

Requirements

• Node.js >= 18.0.0
• element-selector-server running on localhost:8080

License

MIT © Element Selector Team