uia-tree

Turn a Windows GUI into a JSON tree with screen coordinates.

import { uia } from 'uia-tree';

const snap = await uia.readForeground();

const save = snap.find({ role: 'Button', name: 'Save' });
if (save) console.log('click at', save.click);   // { x: 1778, y: 390 }

Reads the Windows UI Automation tree — the accessibility layer screen readers use. Unlike image matching, nothing has to be captured in advance: you ask what is on screen and get an answer, with exact pixel coordinates.

Windows x64 only. The compiled binary ships with the package.

Why this exists

Image matching answers "is this picture present?" — which needs you to know what to look for. This answers "what is on screen?", which is what you need when you don't know in advance. That makes it useful for

• scraping — links with their hrefs, inputs with their values, all the text
• automation — find a button by name, click its coordinates
• recording — turn a click coordinate back into what was there
• agents — hand a model a list of named, numbered, clickable things

It pairs with an input library (such as augui): uia-tree tells you where, the other one does it.

Reading a window

await uia.listWindows();                    // what's open
await uia.readForeground();                 // the focused window
await uia.readByTitle('Chrome');            // first match, or null
await uia.readWindow(handle);               // by handle from listWindows

await uia.elementAt(x, y);                  // just the element under a point
await uia.focusedElement();                 // what has keyboard focus

Options:

await uia.readForeground({
  includeAll: false,    // keep layout containers too (default false)
  maxDepth: 50,
  maxElements: 5000,    // guard against huge trees
  region: { x: 0, y: 0, width: 800, height: 600 },   // only this area
  retries: 3,           // for Chromium's lazy tree — see below
});

What you get

const snap = await uia.readForeground();

snap.window        // "Nasos-NL-811.conf - Notepad"
snap.framework     // "Win32"
snap.tookMs        // 45
snap.elements      // flat, depth-first; elements[0] is the window

Each element:

{
  id: 17,                      // index in this snapshot — not an identity
  depth: 4,
  role: 'MenuItem',            // Button, Edit, Hyperlink, Text, ...
  name: 'File',                // the visible label
  value: undefined,            // input contents; for a link, its href
  automationId: '',            // dev-assigned id — the best thing to match on
  className: '',
  rect:  { x: 1619, y: 406, width: 41, height: 32 },
  click: { x: 1639, y: 422 },  // centre of rect, ready for a mouse move
  framework: 'XAML',
  enabled: true,
  offscreen: false,            // scrolled out of view — NOT hidden
  focusable: true,
  parent: 16,
  children: [],
}

Finding things

snap.find({ role: 'Button', name: 'Save' });
snap.find({ nameContains: 'Save' });
snap.find({ automationId: 'submitBtn' });     // most stable where available
snap.findAll({ role: 'MenuItem' });
snap.findOrThrow({ role: 'Edit' });           // throws instead of returning null

Ready-made views

snap.interactive   // everything clickable/typeable, enabled and on screen
snap.links         // name = link text, value = href
snap.inputs        // Edit and ComboBox, value = contents
snap.text          // all visible text, in reading order

snap.childrenOf(el);
snap.parentOf(el);
snap.pathTo(el);   // root -> ... -> el

snap.toText();     // indented table — fastest way to see what you got
snap.toTree();     // nested object, if you want the shape

Scraping, for real

On Hacker News in a normal Chrome window:

517 elements in 39ms — 229 links

"Mold Linker Version 3.0.0 Release – Rewritten in Rust"
    -> https://github.com/rui314/mold/releases/tag/v3.0.0
"roflcopter69"  -> https://news.ycombinator.com/user?id=roflcopter69
"333 comments"  -> https://news.ycombinator.com/item?id=49917536

INPUTS
  Address and search bar = "news.ycombinator.com"

Titles, authors, comment counts, timestamps — each with its href and its click coordinate. Not the DOM, but for many jobs it is enough, and it comes from the user's own browser with their own session.

const snap = await uia.readByTitle('Hacker News');
const stories = snap!.links.filter((l) => l.name.length > 15 && l.value?.startsWith('http'));
for (const s of stories) console.log(s.name, '->', s.value);

Speed — and why

| window | elements | time | |---|---|---| | Notepad | 37 | 45 ms | | Chrome (Hacker News) | 517 | 39 ms | | VS Code (warm) | ~3000 | ~230 ms |

That speed comes from one thing. Reading UIA properties one at a time costs one cross-process COM call each — 3000 elements × 10 properties would be 30,000 round trips and take seconds. This package bulk-fetches the whole subtree in a single call (CacheRequest), then reads every field in-process.

Fast enough to call per step in a loop.

Chromium needs its accessibility tree woken up

Chrome, Edge and Electron build their accessibility tree only when something asks, and the first read often returns a stub. This is the one real gotcha.

Electron wakes up on its own, progressively. Three consecutive reads of VS Code returned 6 → 241 → 3011 elements. The retries option handles this.

Chrome page content is less willing. Without help you get only the browser chrome — Back, Reload, the address bar — and a Document node with no children. The reliable fix is to launch it with

chrome.exe --force-renderer-accessibility

which costs some CPU for the session, and is exactly why it is off by default.

So: a small tree means "not ready yet", not "nothing there." Read twice before concluding a window is empty.

Gotchas

• id is per-snapshot. It is an array index, not an identity. Re-reading renumbers everything. Match on automationId, or role + name.
• Coordinates are absolute screen pixels, valid only while the window stays put. Move or resize it and they all change — so read fresh rather than caching positions.
• Physical pixels. On a DPI-scaled display, a process that is not DPI-aware will read coordinates in a different space than it clicks in. Worth testing if your machines use scaling.
• value is often empty where an app does not implement ValuePattern.
• Custom-drawn Win32 can collapse to a single opaque Pane — UIA cannot see inside an app that paints its own widgets.
• Text has no per-line coordinates. A text editor gives you one Document with the whole contents in value. Per-character geometry needs the UIA TextPattern API, which this does not expose.
• A locked session renders nothing, but UIA reads over COM rather than from the screen, so it is far less affected than screen capture. Test it directly if unattended runs matter to you.

Playground

In a clone of the source (not part of the npm package):

npm run dump                   # the foreground window, as a table
npm run dump -- --list         # what windows are open
npm run dump -- Chrome         # by title
npm run dump -- Chrome --json  # JSON
npm run dump -- Chrome --all   # include layout containers
npm run watch-clicks           # name whatever is under the cursor

watch-clicks is the germ of a recorder: the hard part of "replay what I did" is turning a coordinate back into a durable description of what was there, and elementAt does that.

Build from source

Needs a Rust toolchain with the x86_64-pc-windows-msvc target.

npm install
npm run build          # release + TypeScript
npm run build:debug    # faster, for iterating

License

MIT

Changelog

0.1.1

• inputs and interactive now include Document. Some apps expose their main text area as a Document rather than an Edit — Windows 11 Notepad has no Edit anywhere, only Document "Text editor". Both helpers previously omitted it, so the only typeable thing in the window was invisible to them with nothing to signal it was missing.
• inputs is now sorted largest first and excludes disabled/offscreen elements. In an editor the main content area is almost always what you mean.