Qualitas Runner (Worker) - Technical Specification

Version: 1.1.0 (Snapshot & History Enabled) Role: Background Worker Service Objective: robust execution of automation flows using Playwright, ensuring thread safety via Snapshots and providing real-time feedback via Redis.

1. Tech Stack

| Category | Technology | Notes | | --- | --- | --- | | Runtime | Node.js v20 (LTS) | Independent process from API. | | Language | TypeScript | Strict typing. | | Queue | BullMQ | Connects to Redis. | | Browser Engine | Playwright | Uses chromium. | | Database Access | Drizzle ORM | Direct DB connection (same config as API). | | Storage | Backblaze B2 | For uploading failure screenshots. | | Real-time | Redis Pub/Sub | Broadcasting progress. |

2. High-Level Architecture

1. Input: Runner receives a Job from BullMQ containing { runId }.
2. Data Source: Runner fetches Snapshot Data (snapshotNodes, snapshotEdges) from the runs table. It does NOT read the live flows table to avoid race conditions.
3. Execution: Iterates through the nodes using Playwright.
4. Broadcasting: Emits events to Redis Channel run-updates:${runId}.
5. Output: Updates runs table with status and stepsResult (JSON).

3. Database Schema Reference

The Worker will interact primarily with the runs table.

// src/db/schema.ts (Reference)

export const runs = pgTable('runs', {
  id: uuid('id').defaultRandom().primaryKey(),
  flowId: uuid('flow_id').references(() => flows.id).notNull(),
  
  // SNAPSHOT DATA (The Source of Truth for this run)
  snapshotNodes: jsonb('snapshot_nodes').notNull(), 
  snapshotEdges: jsonb('snapshot_edges').notNull(),
  
  // EXECUTION STATE
  status: text('status').default('queued'), // running, passed, failed
  environment: text('environment'),
  
  // RESULTS (The Output)
  durationMs: integer('duration_ms'),
  stepsResult: jsonb('steps_result'), // Array of step logs & screenshots
  
  executedAt: timestamp('executed_at').defaultNow(),
});

4. Execution Logic (The Processor)

4.1. Initialization Phase

1. Job Received: Extract runId.
2. Fetch Data: Query runs table where id = runId.

• Validation: Ensure status is queued.
• Update Status: Set status to running.

3. Setup Browser:

• const browser = await chromium.launch()
• const context = await browser.newContext() (Ensures fresh cookies).
• const page = await context.newPage()

4.2. Graph Linearization (Parsing)

The snapshotNodes and snapshotEdges are stored as a graph. You must convert them into an Ordered Execution List.

• Algo: Find the node where type === 'start', then follow the edges to find the next node until the end.
• Output: Step[] (An array of nodes in order).

4.3. The Execution Loop

Iterate through the Step[] array.

For each Step (Node):

1. Resolve Variables: Replace {{VAR}} in data fields with actual values.
2. Broadcast START:

• Redis Publish: run-updates:${runId}
• Payload: { type: "step_start", nodeId: "..." }

3. Execute Action:

• Call Playwright function based on node.type (see Section 5).
• Smart Selector Logic: If primary selector fails, try data.selectors[].

4. Handle Success:

• Push to local executionLog array:

{ "nodeId": "1", "status": "passed", "durationMs": 150, "logs": "Clicked #btn" }

• Broadcast COMPLETE: { type: "step_complete", nodeId: "...", status: "passed" }

5. Handle Failure (Catch Block):

• Capture Screenshot: page.screenshot().
• Upload: Upload to S3/R2 -> Get publicUrl.
• Push to executionLog array:

{ 
  "nodeId": "1", 
  "status": "failed", 
  "error": "Timeout", 
  "screenshotUrl": "https://..." 
}

• Broadcast COMPLETE (Failed).
• BREAK LOOP (Stop execution).

4.4. Finalization Phase

1. Calculate total durationMs.
2. Determine final status: passed (if loop completed) or failed (if loop broke).
3. Update Database:

await db.update(runs).set({
  status: finalStatus,
  durationMs: totalTime,
  stepsResult: executionLog // Save the full history JSON
}).where(eq(runs.id, runId));

4. Close Browser Context.

5. Playwright Action Map

Define a switch-case handler for these node types:

| Node Type | Logic | | --- | --- | | Navigate | await page.goto(url) | | Click | await page.click(selector) | | Type | await page.fill(selector, value) | | Wait | await page.waitForTimeout(ms) | | Assertion | await expect(page.locator(selector)).toHaveText(value) | | Extract | const txt = await page.innerText(selector) (Save to var) |

6. Project Structure (Worker Specific)

src/worker/
├── index.ts                 // Entry point. Connects to Redis/BullMQ.
├── processor.ts             // The main function called by BullMQ.
├── engine/
│   ├── execution-loop.ts    // The "for" loop logic.
│   ├── graph-parser.ts      // Converts Graph -> Array.
│   ├── action-handlers.ts   // Playwright specific commands.
│   └── selector-guard.ts    // Logic for "Smart Selector" retries.
├── lib/
│   ├── storage.ts           // S3 upload wrapper.
│   └── redis-pub.ts         // Wrapper for broadcasting events.
└── types.ts

7. Step-by-Step Implementation Plan for AI

Step 1: Scaffolding

• Create src/worker/index.ts.
• Setup BullMQ Worker to listen to test-queue.
• Mock the processor to just console.log the job data.

Step 2: Graph Parsing

• Implement src/worker/engine/graph-parser.ts.
• Create a unit test: Pass it a JSON graph, assert it returns a correct linear array.

Step 3: The Engine Core

• Implement processor.ts.
• Add logic to fetch Snapshot from DB.
• Implement the Loop (without Playwright first, just mock delays).
• Verify stepsResult JSON structure is built correctly.

Step 4: Playwright Integration

• Implement action-handlers.ts.
• Connect the real Playwright browser.
• Implement "Smart Selectors" (try/catch logic).

Step 5: Failure Handling

• Implement Screenshot capture.
• Implement S3 Mock (or real upload).
• Ensure failed runs correctly update the DB with the error and screenshot URL.

8. Deployment

The Runner is designed to be deployed as a Docker container on a VPS (Virtual Private Server).

8.1. Prerequisites

• Docker and Docker Compose installed on the VPS.
• Access to the NPM Token (if using private registry).
• Redis instance accessible from the VPS.
• Backend API accessible from the VPS.

8.2. Environment Variables

Create a .env file or configure these variables in your deployment environment:

| Variable | Required | Description | | --- | --- | --- | | REDIS_URL | Yes | Connection string for Redis (e.g., redis://user:pass@host:port). | | API_BASE_URL | Yes | Base URL of the Qualitas Backend API. | | WORKER_SECRET | Yes | Secret key for authenticating with the backend (must match API). | | B2_BUCKET_NAME | Optional | Backblaze B2 Bucket Name for storage. | | B2_ENDPOINT | Optional | Backblaze B2 Endpoint. | | B2_KEY_ID | Optional | Backblaze B2 Key ID. | | B2_APPLICATION_KEY | Optional | Backblaze B2 Application Key. | | B2_PUBLIC_URL | Optional | Public URL prefix for uploaded files. |

8.2.1. Pairing-first onboarding (recommended)

For normal usage, you no longer need to manually set Redis/API/queue values.

Install CLI package and pair with a 6-digit code from UI:

npm i -g @qualitas-id/qualinet@latest

# verify installed CLI version
qualinet --version
# or
qualinet -v

# self-hosted user runner
qualinet pair

# managed/internal runner channel (internal-only)
# requires QUALINET_ALLOW_MANAGED_PAIR=true and internal secret
qualinet pair --managed

For managed pairing, internal environments must set:

export QUALINET_ALLOW_MANAGED_PAIR=true
export QUALINET_MANAGED_PAIR_SECRET="<internal managed pairing secret>"

After successful pairing, credentials are stored at ~/.qualinet/runner.json. Security note: self-hosted credentials do not include redisUrl; managed credentials may include it. Managed mode can override Redis via QUALINET_MANAGED_REDIS_URL (preferred). To reconnect later:

qualinet start

8.2.2. Strict security model (required)

• Public npm-installed runners and all customer self-hosted runners are treated as untrusted execution environments.
• Untrusted runners must never receive internal/shared Redis secrets during pairing or bootstrap.
• Self-hosted pairing payloads must not contain redisUrl, and ~/.qualinet/runner.json for self-hosted mode must not persist any Redis credential.
• Managed internal runners may use internal Redis credentials via env-only injection (QUALINET_MANAGED_REDIS_URL preferred, fallback REDIS_URL when explicitly managed).

8.2.3. Redis secret handling by mode

| Mode | Trust level | Redis source | Allowed secret exposure | | --- | --- | --- | --- | | Self-hosted (qualinet pair) | Untrusted | Local operator-provided env (REDIS_URL) | No Redis secrets from backend pairing/bootstrap response | | Managed internal (qualinet pair --managed) | Trusted internal | QUALINET_MANAGED_REDIS_URL first, then REDIS_URL | Allowed only in internal infra controls |

Operational rules:

• Do not copy managed Redis URLs into self-hosted environments.
• Prefer short-lived secret distribution through deployment tooling (not chat, tickets, or static docs).
• Rotate Redis credentials immediately if a managed credential is accidentally exposed to a non-internal host.

8.2.4. Auth token rotation and expiry operations

• Pairing/session auth tokens are expected to expire; runners must re-pair or refresh when backend returns auth-expired/unauthorized responses.
• During planned rotation, run old/new token overlap windows when possible to avoid fleet-wide downtime.
• Rotation order: issue new token -> update runner env/config -> restart runner -> verify successful auth -> revoke old token.
• If token rotation causes repeated auth failures, do not retry indefinitely with stale credentials; stop runner, refresh credentials, and restart.

8.2.5. Replay/idempotency expectations and troubleshooting

• Delivery can be at-least-once; duplicate webhook/update events are possible.
• Runner and backend integrations must treat repeated events for the same runId/step as idempotent and safe to re-process.
• Replays after transient failures should not create duplicate final side effects (duplicate status transitions, duplicate durable writes, duplicate alerts).
• Troubleshoot suspected replay issues by checking: identical runId, repeated step event timestamps, Redis reconnect windows, and backend dedupe logs/constraints.

8.2.6. Verification checklist (concise)

• [ ] Self-hosted ~/.qualinet/runner.json contains no redisUrl.
• [ ] Self-hosted runner only starts when local REDIS_URL is explicitly provided.
• [ ] Managed internal runner prefers QUALINET_MANAGED_REDIS_URL (or approved REDIS_URL) and connects successfully.
• [ ] Token rotation test confirms old token rejection and new token success.
• [ ] Duplicate/replayed event test does not create duplicate durable outcomes.

8.3. Building the Docker Image

You need to pass the NPM_TOKEN as a build argument.

docker build \
  --build-arg NPM_TOKEN=your_npm_token_here \
  -t qualitas-runner .

8.4. Running with Docker

docker run -d \
  --name qualitas-worker \
  --restart always \
  -e REDIS_URL=redis://redis-host:6379 \
  -e API_BASE_URL=https://api.yourdomain.com \
  -e WORKER_SECRET=change_me \
  qualitas-runner

8.5. Deploying with Docker Compose (Recommended)

Create a docker-compose.yml file:

version: '3.8'

services:
  worker:
    build:
      context: .
      args:
        - NPM_TOKEN=${NPM_TOKEN}
    image: qualitas-runner:latest
    restart: always
    environment:
      - REDIS_URL=${REDIS_URL}
      - API_BASE_URL=${API_BASE_URL}
      - WORKER_SECRET=${WORKER_SECRET}
      # storage config (optional)
      - B2_BUCKET_NAME=${B2_BUCKET_NAME}
      - B2_ENDPOINT=${B2_ENDPOINT}
      - B2_KEY_ID=${B2_KEY_ID}
      - B2_APPLICATION_KEY=${B2_APPLICATION_KEY}
      - B2_PUBLIC_URL=${B2_PUBLIC_URL}
    # shm_size is important for Playwright to avoid crashes
    shm_size: '2gb' 

Run the deployment:

# Export the token first
export NPM_TOKEN=your_token

# Build and start
docker-compose up -d --build