@validate.qa/runner

Local test runner for the validate.qa AI testing platform.

This package enables you to execute Playwright and Appium tests generated from your voice narrations directly on your local machine, securely connecting to your validate.qa cloud project.

Features:

• 🎭 Web Testing - Execute Playwright tests in headless Chromium
• 📱 Mobile Testing - Execute Appium tests on iOS Simulator, Android Emulator, AND real devices via USB
• 🔧 Doctor Command - Check system prerequisites for web + mobile testing
• 🔐 Secure Authentication - Project-scoped runner tokens
• 🌐 Tunnel Support - Connect HTTPS dashboards to local devices with token-based auth

Quick Start

1. Install (Global or NPX)

You don't need to install it globally if you use npx, but it's completely supported:

npm install -g @validate.qa/runner

2. Login

Generate a runner token from your validate.qa Project Settings → Local Runner. Your credentials will be saved securely to ~/.validate.qa/credentials.json.

npx @validate.qa/runner login --token urt_xxxxxxxxxxxx --server https://your-server.com

3. Start the Runner

Starts the runner daemon, polling for pending test runs from the cloud and executing them locally:

npx @validate.qa/runner start

Auto-update (required)

Every published runner version can change the client↔server workflow. On login / start / status, and again every ~15 minutes while running, the CLI checks the npm latest dist-tag for @validate.qa/runner. If your install is behind, the process exits with install instructions:

npm install -g @validate.qa/runner@latest
validate-runner start

The server also rejects heartbeats from outdated local runners (HTTP 426) so they cannot claim new work. Cloud / shared runners are not gated by this.

| Env var | Effect | |---------|--------| | VALIDATE_RUNNER_ALLOW_OUTDATED=1 | Emergency bypass (debug / air-gapped only) | | VALIDATE_RUNNER_VERSION_CHECK_MS | Recheck interval while running (min 60s, default 15m) | | RUNNER_CLOUD=true | Skips the check (shared-runner / cloud images) | | MIN_LOCAL_RUNNER_VERSION (server) | Optional floor version if npm is unreachable |

Commands

| Command | Description | |---------|-------------| | login | Saves credentials for future test executions | | start | Starts the local runner (uses saved credentials or CLI flags) | | status | Pings the server to verify your runner token and connection | | logout | Clears all saved runner credentials from your machine | | doctor | Checks system prerequisites for web and mobile testing |

Command Options

login

npx @validate.qa/runner login \
  --token <token> \        # Runner token (urt_...) from dashboard
  --server <url> \         # validate.qa server URL
  --project <id>           # Optional: Project ID (embedded in urt_ tokens)

start

npx @validate.qa/runner start \
  --token <token> \        # Override saved token
  --server <url> \         # Override saved server URL
  --project <id> \         # Override saved project ID
  --mobile \               # Enable mobile testing (Appium + bridge)
  --tunnel \               # Enable ngrok tunnel for HTTPS dashboard access (with token auth)
  --mode <mode> \          # Execution mode: playwright-native (default)
  --poll-interval <ms>     # Poll interval: 2000 (default)

status

npx @validate.qa/runner status
# Shows: Server URL, Project ID, Token (masked), Connection status

doctor

npx @validate.qa/runner doctor
# Checks: Node.js, Playwright, Xcode, Android SDK, Appium

📱 Mobile Testing

Overview

The runner supports native mobile app testing on:

• iOS: Simulators (macOS) + Real devices via USB
• Android: Emulators + Real devices via USB

Tests are executed using Appium with XCUITest (iOS) and UiAutomator2 (Android) drivers.

How It Works

┌─────────────────────────────────────────────────────────────┐
│ 1. START RUNNER WITH MOBILE                                 │
│    npx @validate.qa/runner start --mobile                 │
│    → Starts Appium server (port 4723)                       │
│    → Starts mobile bridge (port 9515)                       │
│    → Discovers available devices                            │
└─────────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────────┐
│ 2. RECORD FROM DASHBOARD                                    │
│    - Open validate.qa dashboard                             │
│    - Start Recording → Select iOS/Android Native            │
│    - Interact with mirrored device screen                   │
│    - Actions captured with screenshots + UI trees           │
└─────────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────────┐
│ 3. GENERATE TESTS                                           │
│    - Stop recording                                         │
│    - Click "Generate from Recording"                        │
│    - Deterministic compiler creates MobileTestStep[] array  │
│    - Test saved with framework: 'APPIUM'                    │
└─────────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────────┐
│ 4. EXECUTE TESTS                                            │
│    - Click "Run" in dashboard (or queue from API)           │
│    - Runner claims mobile test run                          │
│    → Boots simulator/emulator                               │
│    → Launches app via Appium                                │
│    → Executes each step (tap, type, swipe, assert)          │
│    → Captures screenshots                                   │
│    → Reports results back to server                         │
└─────────────────────────────────────────────────────────────┘

Prerequisites

macOS (Required for iOS)

• Xcode - Install from Mac App Store
• Xcode Command Line Tools: xcode-select --install
• iOS Simulator - Included with Xcode

Android (Optional)

• Android Studio - https://developer.android.com/studio
• Android SDK - Installed via Android Studio
• Environment variables: ANDROID_HOME, ANDROID_SDK_ROOT
• Emulator - Create via Android Studio → Device Manager

Node.js

• Version: Node 20 or higher
• npm: Latest stable version

Setup Steps

1. Check Prerequisites

npx @validate.qa/runner doctor

Expected output:

✓  Node.js v20.x.x
✓  Playwright installed
✓  Xcode 15.x (iOS SDK)
✓  Android SDK (API 34)
✓  Appium available

2. Boot a Simulator/Emulator

iOS:

# List available simulators
xcrun simctl list devices

# Boot a specific simulator
xcrun simctl boot "iPhone 15"

# Or open Simulator.app and boot from menu
open -a Simulator

Android:

# List available emulators
emulator -list-avds

# Start an emulator
emulator -avd Pixel_7_API_34

# Or use Android Studio → Device Manager → Start

3. Install Your App

iOS:

# Install .app or .ipa on simulator
xcrun simctl install booted /path/to/YourApp.app

Android:

# Install APK on emulator
adb install /path/to/YourApp.apk

# Or drag-and-drop APK onto emulator window

4. Start Runner with Mobile

For local dashboard (http://localhost):

npx @validate.qa/runner start --mobile

For production dashboard (HTTPS):

npx @validate.qa/runner start --mobile --tunnel

Expected output (with tunnel):

validate.qa Local Runner
─────────────────────────────────────
ℹ  Server:   https://your-server.com
ℹ  Project:  your-project-id
ℹ  Mode:     🎭 Playwright Native + 📱 Mobile 🌐 Tunnel
ℹ  Interval: 2000ms
✓  Browser:  Chrome with user profile
ℹ  📱 Mobile: iOS ✓ | Android ✓ | 2 device(s)
✓  📱 Appium server started
🌐 Public tunnel started: https://abc123.ngrok.io
🔐 Tunnel token: a7f3b2c8d9e1f4a5b6c7d8e9f0a1b2c3d4e5f6a7
   Add this token to your dashboard Project Settings

✓  Runner started - waiting for test runs... (web + mobile)

Note: The tunnel token is automatically sent to your dashboard via the server API. You don't need to manually configure it.

5. Record Your First Mobile Test

1. Open Dashboard → Your Project
2. Click "Start Recording" → Select "iOS Native" or "Android Native"
3. Enter App ID:
   • iOS: Bundle ID (e.g., com.example.myapp)
   • Android: Package name (e.g., com.example.myapp)
4. Select Device from the dropdown (detected automatically)
5. Click "Start" - Device screen mirrors in your browser
6. Interact:
   • Click on screen → Taps recorded
   • Use swipe buttons → Swipes recorded
   • Click "Type Text" → Keyboard input
   • Back/Home buttons → Navigation
7. Click "Stop Recording" when done
8. Generate Tests → Click "Generate from Recording"

6. Run Mobile Tests

The Run Test button queues Appium tests directly. A queued mobile run starts when:

1. You queue a run from the dashboard
2. The runner is online with --mobile flag
3. A matching device is available

The runner will:

• Claim the mobile test run from the queue
• Launch the app on the target device
• Execute each step via Appium
• Capture screenshots after each step
• Report pass/fail results

Runner Modes

Web Testing (Default)

npx @validate.qa/runner start
# or explicitly:
npx @validate.qa/runner start --mode playwright-native

Executes Playwright tests in headless Chromium with:

• Persistent browser profiles (cookies/logins persist)
• Test variable injection
• AI healing support (optional)

Mobile Testing

npx @validate.qa/runner start --mobile

Enables:

• Appium server (port 4723)
• Mobile bridge server (port 9515)
• Device discovery and capabilities reporting
• iOS Simulator + Android Emulator support

Execution Flow Comparison

| Feature | Web Tests | Mobile Tests | |---------|-----------|--------------| | Framework | Playwright | Appium | | Browser/Device | Headless Chromium | iOS Simulator / Android Emulator | | Locator Strategy | CSS selectors, XPath | Accessibility ID, XPath, Resource ID | | Actions | click, fill, navigate | tap, type, swipe, back, home | | Screenshots | On failure | After each step | | AI Healing | ✅ Supported | ❌ Not supported | | Cloud Runner | ✅ Supported | ❌ Local only |

Configuration

Saved Credentials

Credentials are stored in ~/.validate.qa/credentials.json:

{
  "token": "urt_xxxxxxxxxxxx",
  "serverUrl": "https://your-server.com",
  "projectId": "your-project-id"
}

Environment Variables

| Variable | Description | Default | |----------|-------------|---------| | SERVER_URL | validate.qa server URL | - | | AUTH_TOKEN | Runner token | - | | PROJECT_ID | Project ID | - | | POLL_INTERVAL_MS | Poll interval | 2000 | | RUNS_DIR | Directory for test run artifacts | ~/.validate.qa/runs | | CHROME_USER_DATA_DIR | Chrome profile path | Auto-detected |

Example: Custom Configuration

# Run with environment variables
SERVER_URL=https://your-server.com \
AUTH_TOKEN=urt_xxxx \
POLL_INTERVAL_MS=5000 \
npx @validate.qa/runner start --mobile

Troubleshooting

"No devices found"

# iOS: Boot a simulator
xcrun simctl boot "iPhone 15"

# Android: Start an emulator
emulator -avd Pixel_7_API_34

# Then restart runner with --mobile

"App not installed"

# iOS
xcrun simctl install booted /path/to/App.app

# Android
adb install /path/to/App.apk

"Bridge not responding"

# Check if bridge is running
curl http://127.0.0.1:9515/bridge/health

# Restart runner with --mobile flag

"Appium failed to start"

# Check Appium installation
appium --version

# Reinstall Appium drivers
appium driver install xcuitest
appium driver install uiautomator2

# Run doctor to diagnose
npx @validate.qa/runner doctor

"Tests stay PENDING"

# 1. Verify runner is online
npx @validate.qa/runner status

# 2. Check runner is started with --mobile
npx @validate.qa/runner start --mobile

# 3. Verify device is booted
xcrun simctl list devices  # iOS
adb devices                # Android

Security

Runner Token Security

urt_ runner tokens are:

• Project-scoped: They cannot access sessions, user accounts, or cross-project data.
• Revocable: Instantly cancel tokens from the dashboard if compromised.
• Client-only: Your local browser/device instances run locally and post test results back.
• No code upload: Tests execute on your machine - source code never leaves your infrastructure.

Mobile Tunnel Security

When using --tunnel for mobile testing from an HTTPS dashboard:

Security Features:

• Token-based authentication: Each tunnel session generates a random 64-character hex token
• Per-session tokens: Tokens are regenerated every time you restart the runner
• Header validation: All bridge requests require X-Bridge-Token header or tunnelToken query param
• Health endpoint public: Only /bridge/health is accessible without auth (for monitoring)
• All other endpoints protected: Device control, screenshots, and actions require valid token

How It Works:

1. Runner starts with --tunnel
   → Generates random token (e.g., a7f3b2c8d9e1f4a5...)
   → Starts ngrok tunnel to your local bridge
   → Displays token in console

2. Dashboard fetches tunnel URL + token from server API
   → Server receives tunnel info via runner heartbeat
   → Dashboard stores token in memory (never persisted)

3. All bridge requests include token:
   → HTTP header: X-Bridge-Token: a7f3b2c8d9e1f4a5...
   → Or query param: ?tunnelToken=a7f3b2c8d9e1f4a5...

4. Bridge validates token on every request:
   → Invalid/missing token → 401 Unauthorized
   → Valid token → Request processed

Best Practices:

• ✅ Safe for production use: Token is required for all device control operations
• ✅ Token never exposed to client: Dashboard fetches from server API, not console
• ✅ Session-limited: Token changes each time runner restarts
• ⚠️ Don't share console output: The full token is shown in runner console
• ⚠️ Use HTTPS dashboard: Prevents man-in-the-middle attacks on token transmission

For Enterprise Deployments: Consider using a paid ngrok account for:

• Fixed domain URLs (no changing URLs)
• IP whitelisting
• Request inspection
• Higher bandwidth limits

Architecture

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   Dashboard     │────▶│   Server API     │◀────│   Local Runner  │
│  (validate.qa)  │     │  (your-infra)    │     │  (your-machine) │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                                │                        │
                                │                        ├──→ Playwright (web)
                                │                        └──→ Appium (mobile)
                                │                               │
                                │                               ├──→ iOS Simulator
                                │                               └──→ Android Emulator
                                ▼
                        ┌──────────────────┐
                        │   PostgreSQL     │
                        │   (test metadata)│
                        └──────────────────┘

Deploying

GitHub Actions (Automated)

Publishing is automated via GitHub Actions. Push a tag to trigger the workflow:

# 1. Bump version in packages/runner/package.json
# 2. Create and push a tag (format: runner-v<version>)
git tag runner-v0.1.11
git push origin runner-v0.1.11

The workflow (.github/workflows/publish-runner.yml) will:

1. Install dependencies
2. Build shared package
3. Build runner package
4. Publish to npmjs as @validate.qa/runner

Manual (Local)

If GitHub Actions is unavailable:

# 1. Login to npm
npm login

# 2. Navigate to runner package
cd packages/runner

# 3. Bump version (patch, minor, or major)
npm version patch

# 4. Build and publish
npm run build
npm publish --access public

Version

Current: 1.0.36

Changelog: See git history for the full change log. Recent highlights:

• 1.0.36 - Mobile Appium pipeline hardening (cross-platform runs, launch/reset policy, per-test heal, credential redaction)
• 1.0.35 - Force-upgrade gate: outdated installs exit when a newer package is on npm (workflow changes every release)
• 1.0.0 - First stable release
• 0.1.10 - Rename package to @validate.qa/runner; API test capabilities; discovery pipeline refactor
• 0.1.9 - Mobile execution improvements; mobile bridge security hardening
• 0.1.8 - Mobile implementation (Appium 2, device discovery, bridge server)
• 0.1.7 - Security hardening; audio race condition fix; Voice Insights filter fix
• 0.1.6 - Pre-deploy bug fixes
• 0.1.5 - Redesigned local runner UI; publish workflow fixes
• 0.1.2 - Build workflow fix
• 0.1.0 - Initial standalone runner package with secure runner tokens

Support

• Documentation: validate.qa docs
• npm Package: https://www.npmjs.com/package/@validate.qa/runner
• GitHub: https://github.com/Validate-QA/validate.qa

Generated by validate.qa