@qate/cli

Qate CLI for CI/CD pipeline integration. Run test sets and test sequences from your CI pipelines.

Installation

npm install -g @qate/cli

Authentication

Generate an API key from Settings > CI/CD API Keys in the Qate dashboard.

Set the API key as an environment variable:

export QATE_API_KEY=qate_xxxxxxxxxxxx

Or pass it with each command:

qate run -n "My Test Set" --api-key qate_xxxxxxxxxxxx

Commands

Test Sets (Parallel Execution)

Test sets run all tests in parallel for maximum speed.

# List all test sets
qate list:testsets
qate list  # alias

# Run a test set
qate run:testset -n "Regression Suite"
qate run -n "Regression Suite"  # alias

# Run and wait for completion
qate run -n "Regression Suite" --wait

# Run with timeout (default: 600s)
qate run -n "Regression Suite" --wait --timeout 300

# Check execution status
qate status:testset -e ci_xxx_xxx

# Export test set definition
qate export:testset -n "Regression Suite" -o tests.json

Test Sequences (Sequential Execution)

Test sequences run tests in order, optionally sharing browser state between tests.

# List all test sequences
qate list:sequences

# Run a test sequence
qate run:sequence -n "User Flow"

# Run and wait for completion
qate run:sequence -n "User Flow" --wait

# Check execution status
qate status:sequence -e seq_xxx_xxx

# Export test sequence definition
qate export:sequence -n "User Flow" -o sequence.json

Generic Status Command

Auto-detects whether the execution is a test set or sequence based on the ID prefix:

qate status -e ci_xxx_xxx   # Test set
qate status -e seq_xxx_xxx  # Test sequence

Generate Playwright Tests

Generate standalone Playwright test files from your Qate tests. These can be run locally or in your CI pipeline.

# Generate from a test set
qate generate -n "Regression Suite" -o ./e2e-tests
qate generate:testset -n "Regression Suite" -o ./e2e-tests  # explicit

# Generate from a test sequence
qate generate:sequence -n "User Flow" -o ./e2e-tests

# Specify provider (for cloud testing)
qate generate -n "Regression Suite" -o ./e2e-tests --provider browserstack
qate generate -n "Regression Suite" -o ./e2e-tests --provider saucelabs
qate generate -n "Regression Suite" -o ./e2e-tests --provider lambdatest

# Specify browsers
qate generate -n "Regression Suite" -o ./e2e-tests --browsers "chromium,firefox,webkit"

After generating, run the tests:

cd ./e2e-tests
npm install
npx playwright install
npx playwright test

CI/CD Examples

GitHub Actions (Local Playwright Execution)

Run tests locally in your CI using generated Playwright tests:

name: E2E Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Qate CLI
        run: npm install -g @qate/cli

      - name: Generate Playwright Tests
        env:
          QATE_API_KEY: ${{ secrets.QATE_API_KEY }}
        run: qate generate -n "Regression Suite" -o ./e2e

      - name: Install Dependencies
        working-directory: ./e2e
        run: npm ci

      - name: Install Playwright Browsers
        working-directory: ./e2e
        run: npx playwright install --with-deps chromium

      - name: Run Tests
        working-directory: ./e2e
        run: npx playwright test

      - name: Upload Test Results
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: playwright-report
          path: e2e/playwright-report/

GitHub Actions (BrowserStack)

Run tests on BrowserStack's cloud infrastructure:

name: E2E Tests (BrowserStack)
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Qate CLI
        run: npm install -g @qate/cli

      - name: Generate Playwright Tests
        env:
          QATE_API_KEY: ${{ secrets.QATE_API_KEY }}
        run: qate generate -n "Regression Suite" -o ./e2e --provider browserstack

      - name: Install Dependencies
        working-directory: ./e2e
        run: npm ci

      - name: Run Tests on BrowserStack
        working-directory: ./e2e
        env:
          BROWSERSTACK_USERNAME: ${{ secrets.BROWSERSTACK_USERNAME }}
          BROWSERSTACK_ACCESS_KEY: ${{ secrets.BROWSERSTACK_ACCESS_KEY }}
        run: npx playwright test

GitHub Actions (LambdaTest)

Run tests on LambdaTest's cloud infrastructure:

name: E2E Tests (LambdaTest)
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Qate CLI
        run: npm install -g @qate/cli

      - name: Generate Playwright Tests
        env:
          QATE_API_KEY: ${{ secrets.QATE_API_KEY }}
        run: qate generate -n "Regression Suite" -o ./e2e --provider lambdatest

      - name: Install Dependencies
        working-directory: ./e2e
        run: npm ci

      - name: Run Tests on LambdaTest
        working-directory: ./e2e
        env:
          LT_USERNAME: ${{ secrets.LT_USERNAME }}
          LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }}
        run: npx playwright test

GitHub Actions (Sauce Labs)

Run tests on Sauce Labs using saucectl:

name: E2E Tests (Sauce Labs)
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Qate CLI
        run: npm install -g @qate/cli

      - name: Generate Playwright Tests
        env:
          QATE_API_KEY: ${{ secrets.QATE_API_KEY }}
        run: qate generate -n "Regression Suite" -o ./e2e --provider saucelabs

      - name: Install Dependencies
        working-directory: ./e2e
        run: npm ci

      - name: Install saucectl
        run: npm install -g saucectl

      - name: Run Tests on Sauce Labs
        working-directory: ./e2e
        env:
          SAUCE_USERNAME: ${{ secrets.SAUCE_USERNAME }}
          SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
        run: saucectl run

Note: Sauce Labs uses saucectl CLI instead of direct Playwright execution. The generated .sauce/config.yml file contains the test configuration.

GitLab CI

e2e-tests:
  stage: test
  image: mcr.microsoft.com/playwright:v1.40.0-jammy
  script:
    - npm install -g @qate/cli
    - qate generate -n "Regression Suite" -o ./e2e
    - cd e2e && npm ci
    - npx playwright test
  variables:
    QATE_API_KEY: $QATE_API_KEY
  artifacts:
    when: always
    paths:
      - e2e/playwright-report/
    expire_in: 1 week

Azure DevOps

- task: NodeTool@0
  inputs:
    versionSpec: '20.x'

- script: npm install -g @qate/cli
  displayName: 'Install Qate CLI'

- script: qate generate -n "Regression Suite" -o ./e2e
  displayName: 'Generate Playwright Tests'
  env:
    QATE_API_KEY: $(QATE_API_KEY)

- script: |
    cd e2e
    npm ci
    npx playwright install --with-deps chromium
    npx playwright test
  displayName: 'Run E2E Tests'

- task: PublishTestResults@2
  inputs:
    testResultsFiles: 'e2e/test-results.json'
  condition: always()

Jenkins

pipeline {
    agent any
    environment {
        QATE_API_KEY = credentials('qate-api-key')
    }
    stages {
        stage('Generate Tests') {
            steps {
                sh 'npm install -g @qate/cli'
                sh 'qate generate -n "Regression Suite" -o ./e2e'
            }
        }
        stage('Run E2E Tests') {
            steps {
                dir('e2e') {
                    sh 'npm ci'
                    sh 'npx playwright install --with-deps chromium'
                    sh 'npx playwright test'
                }
            }
        }
    }
    post {
        always {
            archiveArtifacts artifacts: 'e2e/playwright-report/**', allowEmptyArchive: true
        }
    }
}

Options

Global Options

| Option | Description | |--------|-------------| | -a, --api-key <key> | API key (or use QATE_API_KEY env var) | | -u, --url <url> | Qate API URL (default: https://api.qate.ai) | | --json | Output results as JSON |

Run Options

| Option | Description | |--------|-------------| | -n, --name <name> | Name of the test set/sequence to run | | --app <id> | Application ID (if name is not unique) | | -w, --wait | Wait for execution to complete | | --timeout <seconds> | Timeout for waiting (default: 600) |

Export Options

| Option | Description | |--------|-------------| | -n, --name <name> | Name of the test set/sequence to export | | -o, --output <file> | Output file path (default: stdout) | | --app <id> | Application ID (if name is not unique) |

Generate Options

| Option | Description | |--------|-------------| | -n, --name <name> | Name of the test set/sequence | | -o, --output <dir> | Output directory for generated files | | --app <id> | Application ID (if name is not unique) | | --provider <provider> | Test provider: local, browserstack, saucelabs, lambdatest (default: local) | | --browsers <browsers> | Comma-separated list of browsers (default: chromium) |

Exit Codes

| Code | Meaning | |------|---------| | 0 | Success (tests passed or still running) | | 1 | Failure (tests failed or error occurred) |

Test Sets vs Test Sequences

| Feature | Test Set | Test Sequence | |---------|----------|---------------| | Execution | Parallel | Sequential | | Speed | Faster | Slower | | State sharing | Independent | Optional | | Use case | Regression testing | User flows, E2E journeys | | Stop on failure | No | Configurable |

Windows Desktop Applications

For Windows desktop applications, tests are executed on a connected desktop agent. The CLI triggers server-side execution and polls for results.

# Run tests on a connected desktop agent
qate run -n "Desktop App Tests"

# Run a sequence on a connected desktop agent
qate run:sequence -n "Desktop Flow"

# Install a build on the agent machine before running tests
qate install --app <applicationId> --path "C:\builds\MyApp-Setup.exe" --args "/silent"

# Full CI pipeline: install build, then run tests
qate install --app 64a... --path "C:\builds\setup.exe" --args "/S"
qate run -n "Smoke Tests" --app 64a...

Install Command Options

| Option | Description | |--------|-------------| | --app <id> | Application ID (required) | | --path <path> | Path to installer on the agent machine (required) | | --args <arguments> | Installer arguments (e.g., /silent, /S) | | --timeout <seconds> | Timeout in seconds (default: 300) |

License

MIT