@avi892nash/aegis-grid-runner

The Aegis WDIO device grid in one self-contained Node bundle — no repo checkout, no node_modules, zero runtime dependencies. Start a runner on any machine and its devices (adb emulators/phones, simctl simulators, installed browsers) are auto-discovered; then add it from the master and they join the grid within ~1s.

The same bundle is the runner, the master, and the job-submitting CLI. aegis-grid-runner with no arguments starts this host's runner (the common case); every other grid subcommand is still reachable.

Pairing model — the master adds runners

A runner never registers itself. It boots unpaired, serves its status page plus a small pairing API on its own port (default 7719), and waits. You then pair it from the master — either the dashboard's Add a runner card or POST /api/runners. The pairing is persisted (GRID_DATA/master.json), so a runner restart re-pairs itself automatically.

runner (unpaired) ── waits ──▶  master  ──POST /api/runners {url}──▶  runner POST /api/master
                                                                           └─ registers, starts leasing

Breaking in 0.5.0: GRID_SERVER self-registration was removed from the runner (it is still the CLI's master URL). A runner started with it logs a warning and stays unpaired until a master adds it.

Why: the two traffic directions have different requirements. runner → master (register, heartbeat, lease, artifact upload) is outbound and works from anywhere. master → runner (job-log tail, browser and Chrome-version management, AVD and simulator control) needs an address the master can route to. Under self-registration the runner advertised a local interface IP it chose itself and the master stored it without ever dialing it — so a tunnelled or cross-network runner sat in the registry at an unroutable address. Jobs ran normally while every control-plane call returned runner unreachable: the fleet read as healthy with half of it dead, and nothing surfaced it until an operator clicked something.

Now the master passes the URL it just reached the runner at, the runner persists it and echoes it back on every register, and the only address that can enter the registry is one that has already been used successfully — including after a restart on either side.

Install & start a runner

# global
npm i -g @avi892nash/aegis-grid-runner
aegis-grid-runner                       # starts UNPAIRED — add it from the master

# or one-shot, no install
npx -y @avi892nash/aegis-grid-runner

Requires Node >= 18. A runner also needs its device tooling on PATH: adb/emulator (Android — see ANDROID_HOME), xcrun/simctl (iOS, macOS only), and/or a locally installed browser (Chrome/Safari/Firefox/Edge) for web + mobile-web.

Add the runner from the master

The runner logs its own address on boot, e.g. runner listening on :7719 — unpaired; waiting for a master to add me.

• Dashboard — open http://<master-host>:7717/ui, expand Runners & devices → Add a runner, type the runner's host:port and click Add.
• API — POST http://<master-host>:7717/api/runners with {"url":"<host>:7719"}. The master hands the runner its own reachable base url (from GRID_PUBLIC_URL, else the request Host header) and the runner registers via the normal flow.

Pairing endpoints served by the runner itself:

| Method | Path | Purpose | | --- | --- | --- | | GET | /api/master | current pairing state | | POST | /api/master {url, runnerUrl?, token?, force?} | pair with a master (409 if already paired elsewhere unless force). A forced switch deregisters from the previous master first, so it cannot keep assigning work to devices that have stopped leasing from it; any job still running across the switch is reported as orphanedJobs because its result belongs to the old master. | | DELETE | /api/master {force?} | unpair (409 if jobs are running unless force; running jobs are left alone) |

The runner's Jobs table and live-log view also expose cancel for every running job. The equivalent API is POST /api/jobs/:jobId/cancel. It stops payload download, dependency installation, or the spawned test process tree (whichever phase currently owns the job), reports CANCELLED to the master, and never consumes the job's retry budget. Master cancellation is also carried on the runner heartbeat, so it works during payload download and install rather than only after the test child starts. If the exact job record disappears or its job-level events cannot be acknowledged for the configured grace, the runner treats the execution as orphaned, kills it locally, frees its slot, and durably reports a cancelled completion instead of waiting indefinitely.

Environment

| Var | Required | Default | Purpose | | --- | --- | --- | --- | | GRID_SERVER | no | — | Ignored by the runner since 0.5.0 (warns if set); pairing is master-initiated. Still the master URL used by the run/CLI subcommands. | | GRID_DATA | no | GRID_WORKDIR | Where the persisted pairing (master.json) is stored. | | GRID_TOKEN | if master set one | — | Shared secret; must match the master's GRID_TOKEN. Also gates this runner's own API. | | GRID_ORPHAN_JOB_GRACE_MS | no | 300000 | Maximum time an active job may go without a successful job-level acknowledgement from its master before the runner cancels it locally. Minimum 30000. Explicit unknown-job responses cancel after GRID_ORPHAN_JOB_MISSES. | | GRID_ORPHAN_JOB_MISSES | no | 2 | Consecutive master 404/410 job-event responses that prove ownership was lost and trigger local cancellation. Minimum 2. | | GRID_BROWSERS | no | chrome=2 | Seeds which browsers are advertised (0 ⇒ off). Since 0.6.0 the number only means on/off — concurrency is a host property, not a per-browser one. | | GRID_BROWSER_BINARIES | no | — | Side-by-side browser aliases and executable paths, e.g. chrome-124=/opt/chrome-124/chrome,chrome-126=/opt/chrome-126/chrome. Use the aliases in GRID_BROWSERS; the runner advertises exact browser-version:* and browser-major:* labels. | | GRID_WEB_SLOTS | no | 2 | Legacy shorthand for the Chrome seed when GRID_BROWSERS is unset (on/off only — see above). | | RESERVE_GB | no | 1 | The only auto knob: RAM to leave the machine. The runner keeps starting browsers while at least this much is free and holds back when it is not. Settable live on the runner page. | | GRID_MAX_CONCURRENT_JOBS | no | — | Replaces auto with a FIXED number of browser jobs at once. Unset ⇒ auto (no count; the RAM reserve governs). | | STARTUP_SETTLE_MS | no | 8000 | How long one browser start is given to show up in free-RAM readings before the next is admitted. | | GRID_AUTO_LIFECYCLE | no | 0 | 1 ⇒ let the runner boot/park local Android emulators on demand (opt-in). | | GRID_PARK_IDLE_MIN | no | 15 | Minutes an idle auto-managed emulator stays up before being parked. | | GRID_MIN_FREE_MB | no | 3072 | RAM floor — an emulator/simulator boot is refused if it would drop free RAM below this. | | PER_SIM_MB | no | 3500 | Estimated RAM per iOS slot — simulator + Appium + WebDriverAgent + the wdio process. Sets how far a pool's parallelism is capped. | | GRID_IOS_SIMULATORS | no | — | Seed simulator pools headlessly: <deviceTypeId>\|<runtimeId>[\|name[\|N]], comma-separated. | | GRID_BUILDS | no | — | Path to a checkout holding builds_android/ + builds_ios/. Optional — the master serves builds, so a runner with none downloads them on demand. | | ANDROID_HOME | Android only | — | Android SDK root; the runner adds platform-tools + emulator to PATH. | | GRID_WORKDIR | no | ~/.aegis-grid | The ONE directory holding ALL runner state (pools, pairing, payload caches, npm cache, Appium drivers, live-view frames). Absolute by default, so npx from any directory lands on the same state. Back-compat: a ./.grid-agent already present in the launch cwd keeps winning. | | GRID_RUNNER_PORT | no | 7719 | Base port for this runner's own status page + pairing/live-log server (scans up if busy). | | GRID_LABELS | no | — | Comma-separated fallback labels advertised when nothing is auto-discovered. | | GRID_JOB_INACTIVITY_MS | no | 600000 (10min) | Output-inactivity watchdog: a RUNNING job that prints nothing on stdout/stderr for this long is killed and FAILED with GRID_JOB_STALLED. 0 disables. Ignored when the job's own --timeout is already tighter. Do not lower this below the longest silence a healthy job can have on your slowest host — a simulator building WebDriverAgent for the first time is silent for the whole build. | | GRID_WDA_WARM_PARALLEL | no | 2 | macOS/iOS. How many WebDriverAgents may build at once on this host. The builds share no work, so extra parallelism only makes each slower: measured on a 24-core/64GB host, a warm takes 0.3–0.4 min at 1–2 concurrent and 16–20 min at 7, with roughly a third failing outright. Waiting leases are held back rather than queued inside the runner. 0 removes the bound. | | GRID_WDA_FAIL_COOLDOWN_MS | no | 600000 (10min) | macOS/iOS. After a FAILED pre-warm, how long before that simulator may spend another full build budget. Without it a degraded instance restarts a fresh 15-minute build on every lease, adding load to the contention that made it fail. The job still builds WDA in-session during the cooldown — only the pre-warm is skipped. | | GRID_WDA_PROBE_MS | no | 5000 | macOS/iOS. Per-attempt cap on the pre-warm's Appium /status poll. Bounds a wedged socket, not the build (that is GRID_WDA_PREWARM_MS); the poll re-runs every second, so this is a miss, not a failure. | | GRID_WDA_TEARDOWN_MS | no | 15000 | macOS/iOS. Per-attempt cap on the pre-warm's session-teardown DELETE. Best-effort — the warm already has what it came for. | | GRID_IOS_MAX_CONCURRENT_JOBS | no | — (off) | macOS/iOS. Optional hard cap on iOS jobs running at once on this host. Deliberately separate from GRID_MAX_CONCURRENT_JOBS, so an existing browser-fleet setting does not silently acquire an iOS meaning. Off by default: GRID_WDA_WARM_PARALLEL is the bound that actually matters. | | GRID_LIVE_SCREEN_W | no | 360 | Width in px of a live simulator frame (see Live simulator view). Smaller is cheaper on the wire; the frame is deliberately low-res. | | GRID_LIVE_SCREEN_INTERVAL_S | no | 2 | Seconds between captures while someone is watching. | | GRID_LIVE_SCREEN_IDLE_S | no | 15 | How long after the last viewer poll the capture keeps running. Must stay comfortably above the UI's refresh interval. |

On the master, GRID_PUBLIC_URL sets the base url handed to a runner when you add it (derived from the request Host header when unset).

Live simulator view

While an iOS job runs, its simulator screen can be watched from the runner page (Jobs → live view) or from the master (Jobs → live), which proxies the frame from whichever runner owns the job. Frames are low-resolution JPEGs (~11 KB) refreshed every couple of seconds — enough to see which screen the test is on and what it is typing.

It is built so that watching costs the runner as little as possible:

• Capture runs in a detached sidecar process, never on the runner's event loop. The HTTP request only streams the last frame off disk, so a slow capture cannot stall another device's lease — the failure mode this runner has hit repeatedly with synchronous forks.
• It uses simctl, not Appium, so watching never touches the test's own WDA session.
• It is pull-only: the capture loop is kept alive by viewers polling, and exits by itself a few seconds after the last one leaves. A host nobody is watching does no work at all.

GET /api/jobs/:id/screen.jpg (runner and master) returns the current frame, 204 while the first capture is still warming, 404 if the job holds no simulator on that runner.

Master-served builds

The job payload deliberately excludes the (multi-GB) builds_android/ + builds_ios/ trees. A runner with a local checkout (or GRID_BUILDS) uses those directly; a runner with none pulls the .apk/.ipa files it needs from the master automatically (background sync while paired, cached under GRID_WORKDIR/builds-cache).

Other subcommands

aegis-grid-runner master start          # run THIS host as the coordinator
aegis-grid-runner run --platform web --range 1-5   # submit a job and wait
aegis-grid-runner devices | jobs | runs # list grid state
aegis-grid-runner status <jobId>        # a job's status (JSON)
aegis-grid-runner runner uninstall      # stop this host's runner + delete its workdir
aegis-grid-runner --version             # build stamp

Uninstall

Everything the runner owns lives in its workdir (default ~/.aegis-grid), so removal is one command:

npx @avi892nash/aegis-grid-runner runner uninstall

It stops the runner processes that own directories in the workdir (each pid is verified via ps to actually be a grid runner before it is signalled), deletes the workdir, and prints what it deliberately leaves behind: simulators/AVDs created for pools (delete via xcrun simctl delete / avdmanager), and npx's own package cache (npm cache clean --force). If the runner was started with GRID_WORKDIR (or with a legacy cwd-relative .grid-agent), run uninstall with the same env / from the same directory. Finally, remove the runner from the master UI — the master keeps showing it offline otherwise.

Browser mode, version comparison, and mobile web

The master run form exposes Headless and Normal (visible) for desktop web/mweb; the selected mode is shown in the runner/device table. Visible mode requires a logged-in GUI session on the runner host.

Browsers: how they may run, and how many at once

Two separate questions, and mixing them is what used to bite. How a browser may run is per version; how many run at once is a property of the machine.

Each browser row is one control with four named states — Off · Headless · Window · Both. Both advertises the same build twice, headless and windowed, as two devices, so one version can serve a CI run and a watch-it-happen run at the same time. A windowed device needs a display; on a display-less host the option is not offered. There is deliberately no per-version parallel count: two versions at 10 each once meant 20 Chromes on one machine, which is exactly the failure a limit should prevent.

Concurrency is one host-wide limit, in the card above the table:

• Auto (default) has no job count at all. The runner keeps leasing while the machine still has RESERVE_GB free and holds back when it does not — no estimate of what a browser "costs" anywhere, because a light page and a heavy one differ by more than 2x and predicting was the mistake. Set the reserve in GB right there.
• Switch Auto off to pin a fixed number instead. GRID_MAX_CONCURRENT_JOBS does the same from the environment and wins over both.
• Only ONE browser start is in flight at a time. A start slot is claimed the moment a lease lands and released once that browser's memory has landed (STARTUP_SETTLE_MS), so N idle devices cannot all read the same free memory and launch together. Jobs the host will not take stay QUEUED on the master for another runner, never parked here.
• If memory never recovers, the job fails on this runner (GRID_LOW_MEMORY) and the master's retry moves it. It is never started anyway — that is what takes a host down.

The runner downloads and manages Chrome for Testing builds itself, no manual install needed. Each added version is its own row beside the host's own chrome.

• Runner page → Browsers → + Add Chrome version opens a searchable list of every known-good CfT build. Adding downloads it (~340 MB, cached). Remove drops the version (409 while one of its Chromes is running a job); the downloaded build stays on disk, so re-adding is instant.
• Master UI → drawer → + Chrome version — all runners adds the same version on every runner at once (per-runner ok/unreachable results shown).
• Headless seeding: GRID_CHROME_VERSION=146.0.7680.165[,151.0.7922.71] registers versions at startup. Choices persist in chrome-versions.json.

Target jobs by version and runner, never by instance:

# any runner, this exact Chrome build
aegis-grid run --requires web-chrome-146.0.7680.165 …
# one runner + one build (the master stamps runner:<id> on every device)
aegis-grid run --requires runner:mac-01-1234 --requires web-chrome-146.0.7680.165 …

Plain web matches any browser; render:<sha> remains the strictest pin (identical rasterization). Keep the same payload SHA, merchant rows, environment, and browser mode when comparing screenshots — text rasterization differs across Chrome builds even with identical fonts, which is exactly why versions are pinnable.

The same Chrome build is also not enough across native macOS and Linux: CoreText and FreeType can rasterize the same downloaded font into different pixels. For a pixel-comparison run, enable Require one verified render class in the master's Advanced section. The master snapshots the selected headless device pool and rejects the run if a browser version or render fingerprint is missing or mixed. Leave it off for ordinary functional runs, whose dynamic cross-runner scheduling is unchanged.

GRID_BROWSER_BINARIES="chrome-124=/opt/cft/124/chrome" still works for builds you manage yourself; version rows are the managed equivalent.

iOS simulator pools

iOS is not shaped like Android and the runner models it differently. An AVD is a pre-created instance — emulator -list-avds is the whole truth and you can only boot what exists. A simulator is a configuration: any (device type × iOS runtime) the host supports can be created on demand. So iOS follows the Chrome-version model, with parallelism on top:

• Runner page → iOS simulators → + add simulator pool opens a searchable list of every device × iOS version this Mac supports. Give the pool a name and an allowed parallel count. Combos the host already has a simulator for are tagged on this host; the rest are created with simctl create when first used.

• A pool named Katalon with parallelism 6 materialises Katalon 1 … Katalon 6. Instances are created lazily — adding a pool of 6 is instant and costs nothing until jobs (or you) start them.

• The pool is one grid device (<runner>-ios-pool-<name>) whose capacity the master enforces at lease time. When a job is leased the runner picks a free instance, creates it if needed, boots it, and hands that job its own IOS_UDID, APPIUM_PORT and WDA_LOCAL_PORT. Jobs target the configuration, never an instance.

• Every instance stays listed with its own ▶ start / ⏻ stop, so you can hold one open to watch it. A manual start counts against the pool's N rather than adding to it.

• RAM has priority over the number you type. The advertised capacity is min(N, what free RAM allows) — the row shows RAM: 2 of 6 with the arithmetic in its tooltip. Nothing is ever killed to make room: a capped pool simply leases fewer jobs and recovers as memory frees up — a pool squeezed to 0 stays registered rather than leaving the fleet. Tune with PER_SIM_MB and GRID_MIN_FREE_MB.

  A slot is budgeted at 3500 MB, measured rather than guessed: two concurrent iPhone 16 jobs took a 24 GB host from ~8400 MB available to a steady ~1300 MB. Be wary of lowering it. Over-admitting is slower than running serially — in that same measurement one of the two jobs blew a 900s timeout while the identical spec alone finished in 3.5 minutes, so the run cost 20 minutes and a retry instead of ~7.

• remove forgets the pool; its simulators stay on disk, so re-adding is instant. Lowering the count behaves the same way — surplus instances stop being leased, they are not deleted.

• Headless seeding: GRID_IOS_SIMULATORS="<deviceTypeId>|<runtimeId>|Katalon|6" (comma-separated for several). Choices persist in ios-simulators.json.

# any runner, this pool
aegis-grid run --requires ios-katalon …
# any iPhone 16 Pro Max simulator, whatever the pool is called
aegis-grid run --requires ios-device:iphone-16-pro-max …

Starting emulators and simulators from the master

Lifecycle controls are no longer confined to each runner's own status page. The master proxies them, so Runners & devices → emulators & simulators on any runner row starts and stops that host's AVDs and simulator instances — including runners you cannot reach directly. ▶ start opens a real window on the runner host when it has a display (auto-lifecycle demand-boots stay headless); on a display-less host the request is downgraded and says so rather than booting something invisible.

Mobile web is a separate platform because it uses Appium and the device browser:

• mweb-android drives Chrome through UiAutomator2 and chooses a matching chromedriver.
• mweb-ios drives Safari through XCUITest. It requires macOS, Xcode/WebDriverAgent, Appium's XCUITest driver, and IOS_UDID (plus IOS_PLATFORM_VERSION when needed).