ai-remote

Pure TypeScript protocol engines (RDP, VNC, SSH) for Node.js, Cloudflare Workers, and web browsers — plus npx ai-remote, a CLI that drives a remote desktop and shows you what it is doing.

🚀 Quick Start (CLI)

Run immediately with npx (zero install). open names the machine once and leaves a session running; every command after it drives that session, so none of them repeats the host, the account or the password:

# Connect and open a live supervision window:
AI_REMOTE_PASSWORD=secret npx ai-remote open 192.168.1.50 -u Administrator

# Take a screenshot:
npx ai-remote shot -o screen.png

# Click or type on the remote desktop:
npx ai-remote click 500,300
npx ai-remote type "notepad"
npx ai-remote key Enter

# Run commands in the remote terminal over SSH:
npx ai-remote exec "dir"

# A host with no desktop -- a Linux box, a Mac, a container -- is a terminal
# session. Keys are used before passwords, so this usually needs nothing set:
npx ai-remote open 192.168.1.60 --ssh -u deploy
npx ai-remote exec "systemctl status nginx"

# Let it go when you are done -- nothing closes on its own:
npx ai-remote close

Agent computer use over VNC and RDP

Open a session once, then reuse it from a persistent Node runtime:

import { RemoteComputer, desktopPoint } from 'ai-remote/computer';

const desktop = new RemoteComputer('my-session');
const image = await desktop.screenshot({ maxEdge: 1000 });
// Inspect the PNG in image.base64, then choose a point in screenshot pixels.
const point = desktopPoint(image, { x: 100, y: 150 });
const after = await desktop.act([
  { type: 'click', ...point },
  { type: 'type', text: 'literal text; including semicolons' },
]);
// after.base64 is the new PNG, ready to return to a vision-capable agent.

The same API drives either protocol. act batches input and returns one in-memory screenshot; dispatch skips capture for a known sequence. Keep the JavaScript handle and your own variables across calls. No CLI subprocess or new remote connection is needed for each action.

For CLI callers, put an action array in a JSON file:

ai-remote batch actions.json --session my-session --shot after.png --json

Actions include repeated clicks (count, optional intervalMs), pointer moves, scrolling, literal text, keyboard chords, waits, and drag paths. See the agent skill for schemas, pacing, and coordinate rules. JSON input is validated before execution, and desktop IPC batches do not interleave with each other.

This supplies remote pixels and input; it does not add browser DOM access, native accessibility trees, or a model. changed and quiet describe observed pixels, not completion of the application's work. Verify results after bursts and observe before choosing actions that depend on a new screen.

What a command costs, and where the time actually goes

Measured on an M4, list with two sessions open:

| | median | | --- | --- | | Node itself, starting and doing nothing | 24 ms | | this CLI, run directly | 27 ms | | npx he --version -- a package with no dependencies | 332 ms | | npx ai-remote list | 328 ms |

The third row is the one that matters. A package with nothing in it costs the same under npx as this one does, because that 300 ms is npm exec resolving the spec, revalidating a packument against the registry and spawning -- work npm does before the package it is running exists to it. No published version of any package can be faster than that row, and this one is at it.

So the way to spend less of it is to invoke npx fewer times, not to make each invocation cheaper. Batch the steps:

# Four npx invocations: ~1200 ms, of which ~1200 ms is npm.
npx ai-remote key MetaLeft
npx ai-remote type notepad
npx ai-remote key Enter
npx ai-remote shot -o s.png

# The same four steps, one invocation: ~305 ms.
npx ai-remote do "key MetaLeft; wait 800; type notepad; key Enter; wait 1500; shot s.png"

ai-remote shell is the same idea for a terminal: one invocation, a session that stays open, and no npm between keystrokes.

Run node bench/cli.bench.mjs in this repository to get the table above for your own machine -- including the no-dependency control, without which npm's 300 ms reads like something this package could fix.

About the npm notice run ... lines

They are npm's, not this package's. @npmcli/run-script writes them unconditionally at npm's default log level:

@npmcli/run-script/lib/run-script-pkg.js:53
    log.notice('run', `${cmd.trim()} ${args?.join(' ')}`.trim())

There is no field in package.json, no .npmrc inside the tarball and no publish flag that reaches them -- they are printed before this program starts. Silencing them is a choice made where the command is run:

npx --silent ai-remote list           # this invocation only
NPM_CONFIG_LOGLEVEL=error npx ...     # this shell only
npm config set loglevel error         # this machine, all npm commands
npm install -g ai-remote              # no npm in the path at all; use `ai-remote`

The last one also skips the 300 ms: ai-remote list answers in about 27 ms. It is a per-machine decision rather than something the package can carry, which is why npx is still what the examples here use.

CLI Commands

  ai-remote open <host[:port]>       connect, show a window, and stay open
  ai-remote open <host> --ssh        a terminal only: no desktop, no RDP
  ai-remote open <host> --vnc        a screen over VNC (implied by port 5900)

  ai-remote shot                     screenshot the desktop
  ai-remote click X,Y                click, --button right|middle, --double
  ai-remote type  "text"             type into whatever has focus
  ai-remote key   <Code>             press keys, e.g. MetaLeft, ControlLeft+KeyA
  ai-remote exec  "cmd"              run a command in the terminal (SSH)
  ai-remote shell                    an interactive terminal
  ai-remote reconnect                a new terminal: fresh login, fresh PATH
  ai-remote clipboard [on|off]       share this machine's clipboard both ways
  ai-remote cp <src> <dst>           copy files; a remote path starts with ':'
  ai-remote do    "steps"            several actions in one go
  ai-remote view                     open the window on the session
  ai-remote view --close             close the window, session keeps running
  ai-remote status                   what the session is doing
  ai-remote list                     every session open, and every password kept
  ai-remote saved                    machines with a password kept
  ai-remote forget [<host[:port]>]   throw one away; --all throws all
  ai-remote close                    close it; --all closes every session
  ai-remote probe [<host[:port]>]    is anything listening?

Sessions

The session opened last is the one the commands drive, and it lives until it is closed -- including across terminals, so an agent and the person watching it reach the same desktop. With more than one open, --session NAME picks between them; a session answers to its name, its host, or host:port.

npx ai-remote open 192.168.1.50 -u Administrator --session desk
npx ai-remote open 192.168.1.60 -u Administrator --session build

npx ai-remote list                  # * marks the one commands will use
npx ai-remote shot --session desk
npx ai-remote close --all

list answers both halves of "what can I get back to". Every open session, with password saved against the ones that have one:

* desk   192.168.1.50:3389  1920x1080  pid 6551  http://127.0.0.1:7373/?t=...  password saved (login keychain)
  build  192.168.1.60:3389  1280x800   pid 6808

Saved, nothing open (`ai-remote open <host>` needs no password):
  [email protected]:3389              login keychain    saved 2026-08-28

The second block is the machines with a password kept but nothing running -- open on one of those needs nothing else. It reads the index beside the keystore rather than the keystore, so a listing never raises a keychain prompt; saved shows that block on its own.

probe is the one command besides open that still takes an address, because it answers whether a session could be opened at all; with no address it probes the current session's host.

A screen over VNC

Macs, Linux desktops and anything else running a VNC server are opened with --vnc, or just by naming port 5900. Everything else is the same as RDP -- the same window, the same shot, click, type, key and do, the same session that stays open until it is closed:

AI_REMOTE_PASSWORD=... npx ai-remote open 192.168.1.70 --vnc -u alex
npx ai-remote shot -o screen.png
npx ai-remote click 640,400

When a Mac offers both account authentication and the legacy VNC-password challenge, -u NAME selects the account exchange and attaches to that user's desktop. Omit -u to use the legacy VNC password instead. Add --save after a successful first connection to keep the password in the macOS login Keychain; later connections can use --no-prompt.

The client is this package's own: no noVNC, no browser and no gateway. It signs in either way a host may ask -- the classic VNC password, or the account and password macOS Screen Sharing wants -- and decodes Raw and CopyRect, which every host implements. Raw is bandwidth-hungry by design; on a LAN a 1080p desktop is fine, over a slow link it will feel it.

A session that is already open can be given a screen without being reopened, which is the point of vnc taking no host:

npx ai-remote open 192.168.1.70 --ssh -u alex    # a terminal
npx ai-remote vnc                                # ...and now a screen too
npx ai-remote exec "uptime"                      # both, one session

Terminal-only sessions

--ssh opens a session that is a terminal and nothing else: no RDP handshake, no framebuffer, and a window that is all terminal. Port 22 implies it, so open box:22 and open box --ssh are the same thing.

exec, shell, status, view and close work as they always do. The commands about pixels -- shot, click, type, key, do -- say there is no desktop rather than failing obscurely.

The session lasts until close. If the host drops the terminal -- an idle timeout on its side, or an exit typed into the window -- the session stays where it is and the next command opens another one; status says disconnected in between.

npx ai-remote open 192.168.1.60 --ssh -u deploy    # a terminal window
npx ai-remote exec "uname -a"
npx ai-remote shell                                # Ctrl-] to leave

Modifier keys between a Mac and a PC

A Mac's shortcut modifier is Command; Windows and Linux use Control. Send the physical key through unchanged and Command-C on a Windows desktop opens the Start menu and types a C, because Command is physically the same key as the Windows logo key.

Microsoft's own Mac client sends it through unchanged and documents the consequence: "The Command key on the Mac keyboard equals the Windows key. To perform actions that use the Command button on the Mac, you will need to use the control button in Windows (for example Copy = Ctrl+C)." That is consistent, and it is also the thing people complain about most, because it asks the fingers to learn a second shortcut modifier for one application.

So by default, when one end is a Mac and the other is not, Control and Command swap:

| pressed on the Mac | arrives on the Windows/Linux desktop | | --- | --- | | Command | Control — Command-C copies | | Control | the Windows key — Control-D shows the desktop | | Option | Alt, unchanged |

and the mirror when a PC keyboard drives a Mac desktop (the VNC case):

| pressed on the PC | arrives on the Mac | | --- | --- | | Control | Command — Control-C copies | | Windows key | Control | | Alt | Option, unchanged |

A swap rather than a redirect, deliberately: it is a bijection, so the Windows key is still reachable and nothing becomes impossible to press. Redirecting Command to Control and leaving Control alone would give a Mac two Controls and no Windows key at all.

npx ai-remote open 192.168.1.50 -u owner                   # adapt, the default
npx ai-remote open 192.168.1.50 -u owner --keys literal     # Microsoft's behaviour

The far end is worked out rather than assumed: RDP is Windows, and for VNC only macOS Screen Sharing offers Apple's Diffie-Hellman security type, which is already what decides how far a scroll notch goes. Ends that agree -- Mac to Mac, PC to PC, Linux to Windows -- are left alone entirely, and status reports what the mapping is doing under keys.

Only the window's live keyboard goes through this. key and do name the key they want on the host, so they are never remapped -- key MetaLeft opens the Start menu on Windows whatever keyboard is in front of you, and key ControlLeft+KeyC means Control-C there. Ctrl+Alt+Del is likewise a sequence the session synthesizes from real Windows codes, not three keys somebody pressed, so the swap does not turn it into Windows+Alt+Del.

The clipboard

On by default. Text copied on this machine reaches the remote clipboard and text copied there reaches this one. The clipboard button in the window's toolbar is the switch, and so is the command line:

npx ai-remote clipboard                       # what it is doing now
npx ai-remote clipboard off                   # stop it for this session
npx ai-remote open 192.168.1.50 -u owner --no-clipboard

Only a desktop carries a clipboard -- RDP negotiates a cliprdr channel and VNC has cut-text in the protocol -- so a terminal-only session has none, the toolbar button does not appear, and nothing is started.

Two things follow from it being a default rather than a choice, and both are worth knowing:

• Everything you copy locally is sent while a session is open. That is the point, and it is also the reason open prints one line saying so. If you are about to copy something the remote machine should not have, clipboard off first.
• It polls, so it costs something. No platform gives Node a clipboard-change event, so the local side is read twice a second for as long as the session is open. Measured on an M4 that is 5.7 ms per read, about 1% of one core per session. The rate stays at twice a second rather than being slowed to save it, because the failure it prevents -- pasting what was on the clipboard before the thing you just copied -- is worse than the cost.

The bridge is the session's, not the page's. The obvious place to put it would be the viewer, using the browser's navigator.clipboard -- but a browser only grants a page the clipboard while that page is focused, and the moment somebody wants the remote clipboard is exactly the moment they have clicked away into a local application to paste it. So the session does the copying, on this machine, whether or not a window is open; the toolbar button turns it on rather than doing it. Two windows on one session are two views of one switch.

Two consequences worth knowing:

• It polls. No platform gives Node a clipboard-change event, so the local side is read twice a second while sharing is on. That is why it is a toggle and not always on, and why off is a real off: the timer is not running and the host's clipboard messages are dropped.
• Nothing goes round in circles. Text that arrived from the host is not then noticed by the poller and pushed back to it, and what was already on the clipboard when sharing started is not treated as a change -- so turning sharing on does not send something copied an hour ago.

It uses whatever the platform has: pbcopy/pbpaste on macOS, Set-Clipboard/Get-Clipboard on Windows, and wl-clipboard, xclip or xsel on Linux. A machine with none reports the clipboard as unavailable and the button stays hidden rather than offering a switch that cannot work.

Sharing sits behind the same control gate as keyboard and pointer input: a watch-only viewer can read the screen and cannot reach either clipboard.

A new terminal, after installing something

A shell inherits its environment when it signs in, and nothing can change a running process's inherited environment from outside it. So a program installed a moment ago is simply not on the PATH of the shell that installed it, and running the command again will not find it however many times it is tried.

reconnect is the fix, and it is the same one a person reaches for -- close the terminal, open another:

npx ai-remote exec "winget install OpenJS.NodeJS"
npx ai-remote reconnect
npx ai-remote exec "node --version"

npx ai-remote exec --reconnect "node --version"   # or both in one call

The session, the desktop and the window stay exactly where they are; only the terminal is new. That also means the working directory and any exported variables are gone, which is the point -- it is a new login, not a refresh.

On Windows a fresh login can still inherit a stale environment, because sshd built its own environment block when the service started. Reading the PATH straight out of the registry works regardless:

npx ai-remote exec '$env:Path = [Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [Environment]::GetEnvironmentVariable("Path","User")'

Copying files

cp is scp with the session standing in for the host: a remote path starts with a colon, and anything else is on this machine.

npx ai-remote cp ./setup.exe :C:/Users/owner/Desktop/   # to the session in hand
npx ai-remote cp :/var/log/syslog ./syslog              # from it
npx ai-remote cp -r ./site :/srv/www                    # a whole directory
npx ai-remote cp desk:/tmp/a.txt build:/tmp/a.txt       # between two sessions

SFTP underneath, which is what scp itself uses since OpenSSH 9 deprecated its own wire protocol. So the host needs no scp binary, a filename with a space or a quote in it is a length-prefixed string rather than a quoting problem, and nothing passes through a shell -- which is why a Windows host needs no special case here even though exec needs several.

Two rules about the grammar:

• A one-letter prefix is a drive, not a session. C:/Users/owner/x.txt is a path on this machine, and always will be. Two or more characters before the colon name a session -- by its name, or by its host.
• A destination that already exists as a directory takes the source's name under it. cp a.txt :/tmp is /tmp/a.txt, and cp -r ./site :/srv/www with www already there is /srv/www/site. A destination that does not exist is the name the copy takes.

Chunks are 32 KB with eight in flight, so a transfer gets the bandwidth the link has rather than one round trip per chunk. The mode travels with the file. The transfer is its own SSH connection rather than the session's, so a large copy does not stall the terminal somebody is watching. --json reports bytes, files and rate.

Signing in

Keys first, then a password.

With no --identity, ssh-agent is asked for whatever it is holding, and ~/.ssh/id_ed25519, id_ecdsa and id_rsa are read if they have no passphrase -- so a host that already takes your key needs nothing set at all. Ed25519, ECDSA and RSA keys are supported, in OpenSSH's own format as well as PKCS#8 and the older PEM containers.

npx ai-remote open box --ssh -u deploy                          # agent + ~/.ssh
npx ai-remote open box --ssh -u deploy -i ~/.ssh/deploy_ed25519 # one named key
ssh-add ~/.ssh/id_ed25519                                       # a key with a passphrase

A key with a passphrase is not read off disk: add it to ssh-agent, which is where a decrypted key belongs, and it is offered from there.

Options for open

• --ssh: A terminal session only, with no desktop (implied by port 22).
• --vnc: A screen over VNC rather than RDP (implied by port 5900).
• -u, --user NAME: Account name on the host.
• -i, --identity FILE: A private key to sign in with. Repeatable.
• -d, --domain NAME: Windows domain or computer name.
• --session NAME: Name this session (default: host_port).
• --ssh-user NAME: Account for the SSH terminal (defaults to --user).
• --ssh-port N: SSH port (default: 22).
• -s, --security MODE: auto | nla | tls | rdp (default: auto).
• -W, --width N / -H, --height N: Desktop resolution (default: 1280x800).
• --idle MINUTES: Close an unused session (default: 0, never).
• --no-clipboard: Do not share this machine's clipboard. Shared by default.
• --keys MODE: adapt (default) swaps Control and Command when one end is a Mac and the other is not; literal sends every key through as itself.
• --headless / --no-view: Run headless without opening the viewer window.
• --no-reuse: A second session beside one that is already open.
• --save: Keep the password for next time, once it has worked.

Options for the rest

• --session NAME: Which session to drive (default: the one opened last).
• -o, --out FILE: Screenshot output file path (default: screen.png).
• -r, --recursive: For cp, copy a directory.
• --reconnect: For exec, open a new terminal first, then run the command.
• --max-edge N: Shrink a screenshot to fit N.
• --json: Output machine-readable JSON results on stdout.

The password, where one is wanted, comes from AI_REMOTE_PASSWORD. There is no --password flag: a command line is visible to every process on the machine.

📦 Library Installation

pnpm add ai-remote
# or
npm install ai-remote

Subpath Exports

| Export | Description | |---|---| | ai-remote | Shared vocabulary (ConnectionSettings, RemoteProtocol, signals, etc.) | | ai-remote/protocols | Driver registry (driverFor, DRIVERS, explainDisconnect) | | ai-remote/protocols/rdp | RDP client, session, credentials, and crypto engines | | ai-remote/protocols/ssh | SSH session, transport, wire protocol, and terminal | | ai-remote/protocols/vnc | VNC session and input management | | ai-remote/shared/* | Direct access to shared models and utilities |

✨ Features

• RDP Protocol Engine:
  • Full X.224, MCS, and GCC capability negotiation.
  • CredSSP v6 with NTLMv2 and TLS 1.2 client implementation over WebCrypto.
  • Fast-path 16-bit interleaved RLE bitmap decoding.
  • Shared clipboard support (cliprdr virtual channel).
  • RemoteApp (RAIL) window orders and execution.
• SSH Engine:
  • RFC 4253 binary packet protocol with curve25519-sha256 / ecdh-sha2-nistp256.
  • AEAD ciphers ([email protected], [email protected]).
  • Terminal interactive and automated agent command execution.
  • SFTP version 3 client on a subsystem channel, for file transfer.
• VNC Engine:
  • RFB protocol integration with Apple Diffie-Hellman handshake support.
• AI Agent Automation:
  • Fast headless actions, structured screen capture, and auto-attached SSH shells.

License

MIT © Leon Guo