testbot-agent

Runs Testbots.ai test scripts on your own machine with a real Playwright-driven Chromium — no Local Agent (Electron/JVM) install required.

It exposes a local HTTP+SSE server on http://127.0.0.1:9201 (modeled on the same contract shape the Electron Local Agent exposes on port 9202 for the main AutomationHQ frontend) for the Testbots.ai frontend to drive.

Install

Requires Node.js 18+ (already installed on most developer/QA machines).

npm install -g testbot-agent

Or run it without installing:

npx testbot-agent start

macOS/Linux (still requires Node.js — Homebrew just wraps the same npm install):

brew install testbots-ai/tap/testbot-agent

Running through your own Node.js — rather than a runtime bundled by us — is a deliberate choice, not just convenience: on Windows, antivirus/EDR software is far more likely to flag Chromium being launched by an unfamiliar, freshly-downloaded runtime than by node.exe, which has years of accumulated reputation from being the single most common thing that launches automated browsers. See Troubleshooting below if Chromium still hangs on launch.

(This repo is private, but that doesn't affect npm publishing — unlike GitHub Release assets, an npm package doesn't inherit its source repo's visibility, so no separate public mirror repo is needed the way GitHub-release-based installs would have required.)

Usage

testbot-agent login <api-token>   # token from Administration → Settings → API Tokens
testbot-agent start               # starts the local runner on 127.0.0.1:9201

Then open the Testbots.ai frontend's test-script Run screen as normal.

start reads a couple of switches from the environment:

| Variable | Default | Purpose | | --- | --- | --- | | TESTBOT_AGENT_PORT | 9201 | Port the local HTTP+SSE server listens on. Change it if something else on your machine already uses 9201. | | TESTBOT_AGENT_HEADLESS | unset (headed) | Set to true to run Chromium headless instead of showing the browser window — e.g. for CI or a machine with no display. |

TESTBOT_AGENT_PORT=9301 TESTBOT_AGENT_HEADLESS=true testbot-agent start

Troubleshooting

If Chromium hangs on launch (common on Windows — antivirus/EDR software intercepting its debugging pipe, or a corrupted download; start's console log will show Launching Chromium... with no further output for up to 2 minutes before failing):

testbot-agent reset-browser

This re-downloads Chromium from scratch and, on Windows, attempts to add a Windows Defender exclusion for %LOCALAPPDATA%\ms-playwright (falls back to printing manual instructions if it can't — e.g. not running elevated, or a different antivirus is active).

Development

Bun is used as the dev/build toolchain (faster installs, built-in bundler) — but the shipped code deliberately avoids any Bun-only API (Bun.serve, Bun.$, Bun.env, etc.) in favor of node:* equivalents, so dist/cli.js runs correctly under plain Node too (see Install above for why that matters).

bun install
bun run src/cli.ts login <api-token>
bun run src/cli.ts start
bun run build          # outputs dist/cli.js (playwright/playwright-core stay external)
node dist/cli.js start # verify it actually runs under Node, not just Bun

Publishing to npm happens in CI (.github/workflows/release.yml) on a tagged release (vX.Y.Z, matching package.json's version) — not via a manual npm publish.

Status

Ported from ahq-actions-commons' ActionLibraryServices (the real execution engine behind test-local-execution-services, which is mostly infrastructure wiring around it) to Playwright/TypeScript. 111 of the ~175 active Java action types are implemented today, organized under src/execution/handlers/ by category — see src/execution/handlers/types.ts for the shared param-extraction helpers every handler uses.

Param key names for every implemented action are confirmed against the live template MongoDB collection (taf-db, verified 2026-08-04) — that collection is the UI-facing step-template registry and the authoritative source for each templateId's real param slot names (via {{placeholder}} syntax in templateTitle). It also settled several Java-source-level ambiguities empirically:

• template-id-189 collided in ActionLibraryServices.java between switchToPreviousWindow (no params) and checkTableRowElement (rowPosition/action-selector params) — resolved at the source: switchToPreviousWindow's @Action annotation was moved to a new, unique template-id-192 in ahq-actions-commons, and a matching template document was added to the live registry (dev only so far, matching the dev-first rollout pattern the recent Table Actions used — needs an ahq-actions-commons release before staging/prod's backend can execute it, but testbot-agent doesn't need that release since it owns its own dispatch table). template-id-189 is now exclusively checkTableRowElement.
• template-id-130 (performNavigationAction) and the malformed template-id-120XX (clearAndSetValue) are similarly absent from the live registry under any ID — both unimplemented, since no real recorded script could ever produce a step targeting them.
• template-id-16/17 (verifyOptionNotPresent/verifyOptionPresent) have a real but incomplete live template — no param exists to specify which option to check for, despite the title implying one. Left unimplemented rather than guessing what "present" would even mean with no target to check.

Supported today:

• Navigation, element interaction (click/fill/check/drag-drop/scroll/hover/keyboard), dropdowns (select/deselect/verify by index/value/visible-text), cookies, HTML table actions (read/click-row-action/check-uncheck-row/verify-row-data), window/tab switching, native alert/confirm/prompt handling, and the bulk of the text/attribute/CSS/geometry/numeric assertion and variable-storage actions.
• API-request steps (v2): status-code assertion, plus property-transfer and JSONPath-style body validation (v1/legacy API steps not ported — v2 supersedes them).
• IF/ELSE branching (condition = element visibility only — comparator-based conditions not yet ported).
• LOOP_COUNT.
• logMessage steps.
• Screenshot capture + upload on step failure, via ahq-background-v2-services' /fileupload endpoint (no new backend work needed — reachable with the JWT this CLI already holds).
• Human-readable step titles in execution results (src/execution/title.ts): templateTitle carries raw {{ui-locator}}/{{text}}/etc. placeholders straight from the MongoDB template collection — previously shown to users verbatim, unresolved. Now substituted with the step's actual locator/text/variable values before being reported as testStepName/testStepTitle.

Paid-plan features (gated on the org's planTier, read from a claim on the login JWT — see src/util/entitlements.ts; a free-tier run never invokes any of this code, not just gets denied after trying):

• SQL querying (template-id-131, src/execution/handlers/db.ts): MySQL/Postgres/SQL Server/Oracle via mysql2/pg/mssql/oracledb (optional dependencies — only whichever driver a given connection actually needs gets loaded), with vault-backed credential resolution against standaloneLocalV2ServiceApiUrl's /rest/api/vault/view/{vaultId} (no auth required, confirmed against the real Java VaultClient). H2 isn't supported (no viable Node driver for Java's embedded engine) and DB2 was dropped from scope entirely — ahq-data-commons' actual DbType enum only defines MYSQL/POSTGRESQL/ORACLE/SQLSERVER/H2, so DB2 was never reachable through this step's real connection model despite appearing in ahq-actions-commons' JDBC driver list. Caveat: template-id-131 has zero real recorded usage across dev/staging/prod, so the assumption that a step's db-connection param embeds the full connection record (matching the ui-locator pattern) rather than a bare id reference is inferred, not confirmed.
• Advanced Function (template-id-135, src/execution/handlers/advanced-function.ts) — all 4 languages the Java engine supports: JavaScript, Python, Java, and Groovy.
  • JavaScript runs in-process via Node's vm module, matching ahq-script-commons' CodeExecutionLibrary.java variable-injection (let name = (<jsonValue>); preamble) and execute()-auto-invocation contract closely — vm.Script's completion-value semantics match GraalVM's context.eval() closely enough that this isn't an approximation for JS specifically.
  • Python and Groovy shell out to a local python3/groovy (only if found on PATH — probed lazily, cached, and only ever for an already-confirmed-paid run per the tier-gating rule) and capture an auto-invoked execute() function's return value via a marker-delimited stdout line; a script that calls execute() itself (rather than defining it and letting testbot-agent auto-invoke) falls back to full captured stdout as the result — subprocess execution has no analogue to "take the return value of an arbitrary function call," unlike GraalVM's in-process eval. Groovy's variable injection uses core JsonSlurper/JsonOutput (always present, no extra dependency), so unlike Java below it isn't limited to scalar values.
  • Java compiles+runs via a local javac+java, following the documented "agreed contract" in ahq-actions-commons' JavaCodeProcessor.java: the user writes a function body only (no public static void main), wrapped as getCodeSnippetFunction(). Two real, documented simplifications: variable injection is scalar-only (string/number/boolean) since Java's var name = <literal>; has no array/object literal syntax without a JSON library on the user's ad-hoc classpath; and the more flexible input shapes JavaCodeProcessor also accepts (a full pasted class, a differently-named method) exist for the Monaco editor's autocomplete UX, not the core execution contract, and aren't replicated.
  • All four languages are tested against real interpreters (JS 8 scenarios, Python 6, Java 6, Groovy 7 — last-expression/return value, auto-invoke, no-double-invoke, stdout fallback, variable injection, runtime errors), all passing, including a real bug the Groovy testing caught: variable injection there uses bare assignment (name = ..., not def name = ...) — a top-level def becomes a local variable of Groovy's implicit run() method, invisible from a separately-defined execute() method (the primary expected script shape), and threw MissingPropertyException until fixed.
  • Explicit tradeoff: subprocess execution (Python/Java/Groovy) has no sandboxing at all — unlike GraalVM's isolated Context or even Node's vm module (partial isolation for JS) — acceptable since these scripts are authored by the org's own users, same trust model the Java engine already assumes for compiled-Java/Groovy steps.
• Selenium/Appium dual driver stack (src/execution/selenium/) — a genuinely separate execution path from the Playwright stack above, dispatched by src/execution/dispatch.ts based on planTier and loaded via a dynamic import(), not a static one: webdriverio's dependency tree is large enough that a static import bloated dist/cli.js from ~0.65MB/78 modules to ~16MB/1300+ modules, and would have made webdriverio a hard runtime requirement for every user, free tier included, even though it's an optionalDependency. Dynamic import means free-tier users never load it, and it isn't in their bundle at all — confirmed by testing both branches of the dispatcher directly.
  • Desktop coverage is essentially complete: 110 of ~131 in-scope actions, full IF/ELSE/LOOP_COUNT control flow, window/tab management, native alert handling, and screenshot-on-failure. selenium-runner.ts's executeSeleniumSteps deliberately duplicates runner.ts's executeSteps shape rather than sharing/refactoring it — zero regression risk to the working, tested Playwright path, at the cost of some duplication. Every ported action reuses the SAME param keys already confirmed against the live template MongoDB collection for its Playwright equivalent — no new param-key research needed, since the wire contract doesn't change per driver.
  • Alerts are genuinely simpler than the Playwright version, not just ported: Selenium/WebdriverIO's alert API (getAlertText/acceptAlert/dismissAlert) is blocking and talks directly to whatever native dialog is open — no dialog-listener/auto-accept-safety-net architecture needed the way Playwright requires.
  • Window management needs no client-side "current page" tracking: WebdriverIO's session tracks the focused window server-side (browser.switchToWindow() is real session state), unlike Playwright's independent Page objects — the only client-side state needed is a single previousWindowHandle string for switchToPreviousWindow (template-id-192, the same live-in-dev-only templateId from the Playwright-side fix earlier). Finding a window by title/locator still requires switching to each candidate temporarily to inspect it, since Selenium can't read another window's content without focusing it first.
  • Screenshot-on-failure shares its upload logic with the Playwright stack directly (screenshot.ts's uploadScreenshotBytes was extracted and is called by both — the only driver-specific part is capturing the PNG bytes in the first place: page.screenshot() vs. browser.takeScreenshot(), which returns base64 rather than raw bytes).
  • template-id-181 (API request) and template-id-182 (logMessage) are fully driver-agnostic; the API handler's core logic is exported and shared directly with the Playwright stack, not duplicated.
  • Reuses report-client.ts's reporting types/function and title.ts's resolveDisplayTitle directly (both fully driver-agnostic already) rather than duplicating them.
  • Local driver resolution has a real fallback: WebdriverIO's default driver auto-resolution (Selenium Manager) hits public vendor CDNs — confirmed to genuinely hang (not just fail fast) against a real local Chrome session on a network with restricted outbound access, not a hypothetical concern. src/execution/selenium/driver-download.ts falls back to AutomationHQ's own asset CDN instead — the same S3+CloudFront-backed, unauthenticated manifest system (storeUrl, already in every JWT) the Electron Local Agent already uses for exactly this (chromedriver/geckodriver/edgedriver/appium binaries with per-platform URLs + sha256 checksums). driver-factory.ts races the default path against a 20s timeout, then downloads+extracts+spawns the matching driver locally and connects directly to it, remembering the fallback was needed for the rest of that start session so later runs don't repeat the timeout.
  • Testing footprint: the checksum-verification, zip/tar.gz extraction, recursive executable-search (including the real per-vendor binary-name gap — Edge's asset id is "edgedriver" but its actual binary is "msedgedriver"), and driver-server spawn/readiness/timeout logic are all verified against synthetic archives/binaries.
  • Live-verified end-to-end (2026-08-07), both Debug Mode and Local Bot Execution Mode, against a real script on dev.automationhq.ai (RealtyVista - Agent Login, run via TESTBOT_AGENT_TESTING_MODE=true): real storeUrl manifest fetch → real chromedriver download+launch → WebdriverIO connecting to it → full 6-step login flow, all PASSED, screenshot-on-failure path also exercised. Local Bot Execution Mode's per-step/per-suite incremental reporting was independently confirmed by reading the resulting testScriptResults/testSuiteResults documents back out of MongoDB, not just trusting the CLI's own success message. Found and fixed one real, load-bearing bug in the process (present on both drivers, not new to Selenium): resolveLocator/resolveSeleniumLocator's own existence-probe used a hardcoded 2000ms timeout regardless of caller — template-id-36/37 (wait for displayed/clickable, with a user-configured wait-duration param) and 95 (verify present, 10s bound) all had their real, intended wait duration silently capped at 2s, since the probe threw before the handler's own longer wait ever ran. Both now thread the real intended timeout through instead of relying on the default; resolveLocator/resolveSeleniumLocator gained an optional timeoutMs parameter (defaulting to 2000, unchanged for every other caller) to make this possible.
• Remote grid support (src/execution/selenium/remote-driver-factory.ts) — BrowserStack/LambdaTest/Selenoid/TestingBot/generic Grid, dispatched off an optional gridId on the run request. Fetches the grid record from GET {standaloneLocalV2ServiceApiUrl}/rest/api/grid[/{gridId}] (confirmed against StandaloneLocalController.java — an earlier research pass had this pointed at ahq-config-services' plural /rest/api/grids/{gridId} instead, which actually backs the out-of-scope cloud-side Remote Bot Execution Mode; corrected after direct challenge), extracts credentials from the hub URL's userinfo (falling back to separate username/accessKey fields), detects the vendor by substring match on the hub URL (matching ChromeFactory.java's own approach, not a dedicated Grid.type field), and builds the matching vendor capabilities object (tb:options/LT:Options/bstack:options/selenoid:options/moon:options). Fails soft to a local driver on any grid-lookup error, mirroring BotExecutionRepository's real "assume local execution" fallback. Live-tested against a real TestingBot grid and blocked by the vendor, not this code: a raw curl POST to the identical TestingBot session-creation endpoint with the identical credentials also hung with zero response (while a simple status-endpoint GET on the same host succeeded instantly), proving the request this code sends is correctly formed — the hang is on TestingBot's account side (likely quota/stale credentials), not verifiable further without different grid credentials.
• Mobile/Appium actions (src/execution/selenium/handlers/mobile.ts, mobile-driver-factory.ts) — all 31 actions (template-id-137 through 167): tap/double-tap/long-press (element and coordinate-based), swipe gestures, scroll-to-text, app lifecycle (launch/close/reset/install/remove/is-installed), device actions (orientation, lock/unlock, hide keyboard, back button), element/text/toast verification, attribute reads, and native/webview context switching. Session creation resolves an optional app version id to a real S3 fileUrl (GET {testManagementServiceApiUrl}/rest/api/mobile-versions/{id}) and sets it as the appium:app capability. Carries over the real, hard-won "Facebook Lite" fix from ActionLibraryServices.java: mobileLaunchAndroidApp/mobileLaunchIosApp refuse to silently report success when no real app is configured, rather than accidentally activating the home-screen launcher. Flagged untested, not just unverified: no Appium server, Android emulator, or iOS simulator is available in this environment (confirmed via which appium/which emulator/xcrun simctl list devices) — every line here is typecheck/build-verified only, per an explicit "build it now, flag as untested" decision rather than silently deferring or silently shipping it as if it were tested. Two real, narrow capability gaps found while porting, not silently dropped: mobileIsAppInstalled/mobileGetElementAttribute have no "store as variable" param in the live template collection (Java stores their result via a status.returnValue channel this CLI's step-handler contract has no equivalent for) — their result is logged to the CLI's own stdout instead of silently discarded, but isn't chainable into a later step or visible in the SSE/UI stream.
• LOOP_EACH (src/execution/loop-each.ts, shared by both drivers since collection resolution touches no page/browser) — ported now that the Java engine's own implementation matured well past the "young/fragile" state that originally justified deferring this (CommonTestExecutionService.java's executeLoopStep/resolveLoopEachCollection now has a full, documented three-mode contract). Supports LIST (a literal array authored in the step) and CODE (an Advanced Function snippet — JS/Python/Java/Groovy, reusing Track B's existing sandboxes directly — whose return value is coerced to an array) collection modes, plus the legacy free-text/Variable path (a variable already holding a JSON-encoded array). Deliberately does not support UI_LOCATOR mode (iterate over live DOM/native elements found by a locator): Java resolves each iteration's item as an actual WebElement handle that a later step's locator param binder reads back out of thread-local state, and testbot-agent's ctx.variables is a scalar Map<string, string> with no live-element-handle concept and no seam for a step's locator param to be resolved dynamically from a variable — attempting it would mean either a silent wrong behavior or a much larger architectural change, so it throws a clear, specific error identifying the gap instead. Inferred from the Java source and the CollectionValuePair/Params entity shapes, not confirmed against a real recorded step — same "inferred, not confirmed" caveat the db-connection param carried before it turned out to be wrong; flag loudly if a real LOOP_EACH script's wire shape doesn't match.

Not yet ported (tracked as follow-up work, not permanently excluded):

• Comparator-based IF/ELSE conditions (text/number operators, AND/OR chains) — condition steps only check element visibility today.
• LOOP_EACH's UI_LOCATOR collection mode (see the LOOP_EACH bullet above for why).
• Pre-scripts and nested CommonFunction/subScriptId recursion (the same underlying mechanism in Java).
• Frame/iframe switching — Playwright's locator model doesn't map onto Selenium's "switch into a frame" the way window/tab switching does; needs its own design pass.
• template-id-130 (performNavigationAction), template-id-16/17 (verifyOption{Not}Present), and the malformed template-id-120XX (clearAndSetValue) — all confirmed absent (or, for 16/17, unusably incomplete) in the live template registry; see src/execution/handlers/interaction.ts and dropdowns.ts for details.