DevAssure Invisible (QA) Agent CLI

Command-line interface for DevAssure Invisible (QA) Agent - executes end to end UI tests from natural language instructions and csv files. CLI can be used to run tests by integrating into CI/CD pipelines.

Prerequisites

• Node.js 18+

Installation

Install DevAssure CLI globally using npm:

npm install -g @devassure/cli

Or using pnpm:

pnpm add -g @devassure/cli

After installation, verify it's working:

devassure version

Getting Started

1. Login

Authenticate with DevAssure:

devassure login

This will open your browser for OAuth2 authentication.

Alternatively, you can add an authentication token directly (recommended for CI/CD pipelines):

devassure add-token <your-token>

2. Initialize Your Project

Initialize DevAssure configuration in your project:

devassure init

This will:

• Create a .devassure folder with configuration files
• Prompt you for app URL, description, and personas
• Create a sample test file

3. Add Test Cases

Add test cases to the project by adding yaml files to the .devassure/tests folder in the project directory.

Example test case file:

summary: Sample test case
steps:
  - Open the application url
  - Verify if the page loads successfully
  - Verify if there are no error messages
priority: P0
tags:
  - sanity
  - app-load

4. Run Tests

Run tests using the current directory as test cases directory:

devassure run-tests
# or use the alias
devassure run

5. Add Test Data

Any required test data can be added to .devassure/test_data.yaml in the project directory.

Example test data file:

default:
  url: 'http://localhost:3000'
  users:
    default:
      user_name: '[email protected]'
      password: 12345678
    admin:
      user_name: '[email protected]'
      password: 12345678

Commands

Authentication

• devassure login - Login to DevAssure using OAuth2 authentication
• devassure logout - Logout from DevAssure and clear stored tokens
• devassure add-token <token_value> - Add and validate an authentication token

Configuration

• devassure init - Initialize DevAssure configuration in the current directory
  • Creates .devassure folder with configuration files
  • Prompts for app URL, description, and personas
  • Creates sample test file

Running Tests

• devassure run-tests - Run tests using current directory as test cases directory

  • --path <path> - Project path (default: current directory)
  • --csv <path> - Path to CSV file containing test cases (relative or absolute). Relative paths in order: current directory, .devassure are supported. File must exist and have .csv extension.
  • --tag <tags> / --tags <tags> - Comma-separated tag values (e.g., --tag=tag1,tag2,tag3)
  • --priority <priorities> / --priorities <priorities> - Comma-separated priority values (e.g., --priority=P0,P1)
  • --folder <folders> / --folders <folders> - Comma-separated folder paths (e.g., --folder=admin/users,project/integration)
  • --query <query> / --queries <query> - Search query string (e.g., --query="my search query")
  • --filter <filter> / --filters <filter> - Raw filter string (takes precedence over other filter parameters, e.g., --filter="tag = tag1,tag2 && priority = P0")
  • --archive <folder> - Archive folder path; after the run, test reports are written as devassure-results-<session-id>.zip in this folder (relative paths are resolved from current directory)
  • Note: If --filter/--filters is provided, all other filter parameters (--tag, --priority, --folder, --query) are ignored
  • Alias: devassure run (same options apply)

• devassure resume - Resume an existing test session by session ID

  • --session-id <session-id> - Session ID to resume (one of these or --last is required)
  • --last - Resume the last executed session

Reports and Statistics

• devassure archive-report - Archive report results for a test session into a zip file

  • --output-dir <dir> - (Required) Output directory for the archived report zip
  • --session-id <sessionId> - Session ID to archive (either this or --last is required)
  • --last - Use the last executed session ID from the database
  • Runs devassure-agent archive-results and writes devassure-results-<session-id>.zip into the output directory

• devassure open-report - Open report server for a test session

  • --session-id <sessionId> - Session ID to open report for
  • --last - Open report for the last executed session
  • --archive <path> - Path to an archived zip file (e.g. from archive-report) to open; opens the report from the zip without using the database
  • Note: One of --archive, --session-id, or --last is required

• devassure summary - Print summary for a test session

  • --session-id <sessionId> - Session ID to summarize (either this or --last is required)
  • --last - Use the last executed session ID from the database
  • --json - Output summary as JSON (session_id, environment, scenarios, score, grouped_failures, passed_validations, duration_ms, duration_string)

• devassure stats - Show statistics about sessions, scenarios, and storage

  • Displays test session count, scenario count, and storage usage

Maintenance

• devassure cleanup - Clean up execution history to free up space
  • --retain-days <days> - Retain sessions from the last N days
  • --retain-sessions <count> - Retain the last N sessions
  • If no options provided, prompts to delete all sessions

Utility

• devassure version - Show version of the CLI
• devassure help - Show help information

Global Options

• --verbose - Enable verbose logging for all commands

Configuration

Project files files are stored in:

• ~/.devassure/ (project-specific)

Examples

Basic Workflow

# 1. Login
devassure login

# 2. Initialize project
devassure init

# 3. Run tests
devassure run-tests

# 4. View report for last session
devassure open-report --last

Cleanup

# Keep only sessions from last 7 days
devassure cleanup --retain-days 7

# Keep only the last 10 sessions
devassure cleanup --retain-sessions 10

Filtering Tests

# Run tests with specific tags
devassure run-tests --tag=smoke,regression

# Run tests with specific priority
devassure run-tests --priority=P0,P1

# Run tests in specific folders
devassure run-tests --folder=admin/users,project/integration

# Run tests with a search query
devassure run-tests --query="login flow"

# Combine multiple filter parameters
devassure run-tests --tag=smoke --priority=P0 --folder=admin/users

# Use raw filter string (takes precedence over other filter parameters)
devassure run-tests --filter="tag = tag1,tag2 && priority = P0"

# Use keyword search to filter tests (this will do text match on summary, steps, tags)
devassure run-tests --filter="query = dashboard"

# All parameters support plural forms
devassure run-tests --tags=smoke,regression --priorities=P0,P1 --folders=admin/users

# Resume the last session
devassure resume --last

# Resume a specific session
devassure resume --session-id=<session-id>
devassure resume --id=<session-id>

# Run tests and archive reports to a folder (zip created after run completes)
devassure run-tests --archive=./reports
devassure run --archive=/tmp/archives

# Run tests from a CSV file (relative path resolved from project path)
devassure run-tests --csv=sample-tests.csv
devassure run --csv=.devassure/sample-tests.csv

Archiving and Opening Reports

# Archive the last session's report to a directory
devassure archive-report --output-dir=./reports --last

# Archive a specific session's report
devassure archive-report --output-dir=./reports --session-id=<session-id>

# Open report from an archived zip file
devassure open-report --archive=./reports/devassure-results-<session-id>.zip

Session Summary

# Print summary for the last session
devassure summary --last

# Print summary for a specific session
devassure summary --session-id=<session-id>

# Output summary as JSON
devassure summary --last --json

Sample configuration files

All configuration files live under .devassure/. Below are sample contents for each file type.

app.yaml

App description and rules for the agent:

description: >
  About my app
  in multi line
rules:
  - User can't create more than 3 projects
  - Any data deleted can't be restored

personas.yaml

(Optional) User personas used in tests:

normal_user:
  description: No admin access can do all other operations
  age_group: 18-30
  gender: M
  region: USA
admin:
  description: Admin can add or delete other users and manage privileges
deactivated_user:
  description: Login is deactivated by admin
expired_user:
  description: License has expired

test_data.yaml

Default URL, users, and test data per environment:

If data keys are not present for a specific environment, the default data will be used.

default:
  url: 'http://localhost:3000'
  users:
    default:
      user_name: '[email protected]'
      password: 12345678
    admin:
      user_name: '[email protected]'
      password: 12345678
  stripe_data:
    allowed_card:
      card: 4444 4444 4444 4444
    wrong_cvv:
      card: 1234 1234 1234 1234
      expiry: 10/28
      cvv: 123
  test_otp: 1122
uat:
  url: 'http://uat.myapp.com'
  users:
    default:
      user_name: '[email protected]'
      password: 12345678
    admin:
      user_name: '[email protected]'
      password: 12345678
  test_otp: 2345

agent_instructions.yaml

Instructions for the agent during test execution:

instructions:
  - Reload the app and retry if app shows warning server is busy
  - Sign up a new user and proceed if any of the given logins are not working

preferences.yaml

Browser and execution preferences:

browsers:
  default:
    browser: chromium
    resolution: 1920 x 1200
    headless: true
workers: 2

csv_mapping.yaml

Column mappings when running tests from a CSV file (used with --csv). Maps CSV column names to test case fields:

summary: Summary
steps: Steps
priority: Priority
tags: Tags

Sample test cases

Example test case in .devassure/tests/sample-test.yaml:

summary: Sample test case
steps:
  - Open the application url
  - Verify if the page loads successfully
  - Verify if there are no error messages
priority: High
tags:
  - sample
  - ui

Another example with multiple steps:

summary: Login as admin and verify dashboard
steps:
  - Open the application url
  - Log in with admin credentials from test data
  - Verify dashboard loads and shows admin menu
  - Log out
priority: P0
tags:
  - smoke
  - login
  - admin

Actions

Actions group steps that can be reused across multiple tests. Add action YAML files under .devassure/actions/. To call an action, use its name as a step in a test file.

Fields in an action file:

| Field | Description | |-------|-------------| | name | The name of the action. Used to call the action from the test file. | | description | A short description of the action. | | steps | A list of steps that make up the action. |

Example action file (e.g. .devassure/actions/login_as_admin.yaml):

name: login_as_admin
description: Login to the app as admin using google
steps:
  - Open admin portal url
  - Click on Google login button
  - Enter admin email and password
  - If MFA is asked, enter the authenticator OTP
  - If allow access is asked, click on allow access

Example 2 (e.g. .devassure/actions/add_manager_user.yaml):

name: add_manager_user
description: Add a manager user to the app
steps:
  - Go to users page
  - Click on Add User button
  - Enter email
  - Enter role as Manager
  - Once team dropdown appears assign any team to the user
  - Enter random user details
  - Click on Add User button

Usage in test file: reference actions by name in steps:

steps:
  - login_as_admin
  - add_manager_user
  - Open users list page
  - Verify if the manager user is added

Library tools

Library tools are inbuilt tools that users can include in the project by listing the required tools in a library.yaml file.

Sample library.yaml (e.g. .devassure/library.yaml):

tools:
  - 'authenticator'
  - 'faker:*'

faker

Faker provides realistic synthetic data generators for tests:

| Tool-key | Description | |------|-------------| | *first_name* | Generate a random first name | | *last_name* | Generate a random last name | | *full_name* | Generate a random full name | | *email* | Generate a random email address | | *phone* | Generate a random phone number |

authenticator

Authenticator provides TOTP (Time-based One-Time Password) helpers:

| Tool-key | Description | |------|-------------| | get_authenticator_otp | Generate a TOTP code from an authenticator secret. The authenticator secret must be passed to generate the code. |

Tools

Tools let you run custom commands or programs and consume their output. The configuration lives in .devassure/tools/index.yaml.

Requirement: tools/index.yaml is required for tools to work. It must exist at .devassure/tools/index.yaml (relative to your project path).

Tools can run any command or program; the runner executes the command and consumes its output. For custom code or scripts, put the tool code inside the .devassure/tools folder and reference the execution command in tools/index.yaml (e.g. exec: node script.js or exec: npm run myTool).

Note: You can use any programming language or program. You are responsible for setting up dependencies (e.g. Node.js, Python, venv, system binaries).

Mandatory fields

Each tool must have name, description, and exec:

| Field | Description | |-------|-------------| | name | Unique identifier for the tool (e.g. used when invoking the tool). | | description | Short description of what the tool does. | | exec | Command(s) to run. Can be a single line or a multi-line string. Supports ${argName} substitution for args. |

Optional per-tool fields include cwd and args (see below).

Minimal example:

tools:
  - name: "getProjectDetails"
    description: "Get project details from api"
    cwd: "api-tools"
    args:
      - name: projectId
        type: string
    exec: npm run getProjectDetails ${projectId}

• cwd (optional): Working directory for the command. It is relative to the tools folder (.devassure/tools). If omitted, the working directory is the tools folder. Absolute paths are also supported.
• args (optional): List of parameters. Each has name and type; see Supported arg types and optional args below.

Optional top-level configuration: settings and setup

settings: Default options applied to every tool run and every setup step. For all fields except env, the value on the tool or setup step is used when present; values in settings are fallback when the tool or step does not define that field. For env, the list of environment variables is merged: settings env vars and the tool/setup step env vars are combined (tool/setup entries override settings when the same key is used).

setup: Optional list of steps run once per session before scenario executions start (e.g. install dependencies). Each step can specify name, cwd, exec, and any of the settings fields; inheritance from settings applies when a step omits a field.

cwd behavior

• Relative: Resolved from the tools folder (e.g. cwd: "api-tools" → .devassure/tools/api-tools).
• Default: If cwd is not provided, the working directory is the tools folder.
• Absolute: Absolute paths are supported (e.g. cwd: "/opt/scripts").

Settings and tool/setup options

| Option | Applies to | Description | |--------|------------|-------------| | timeoutSec | Tool / setup | Maximum execution time in seconds (e.g. 10). | | output.start / output.end | Tool / setup | Markers in stdout; only content between these markers is captured as the tool output. Strongly recommended when the process prints a lot of extra content (logs, progress). Your script should print the actual result between these markers. | | env | Tool / setup | List of KEY: value environment variables added or overridden for the process. | | ignore_failure | Tool / setup | If true, a non-zero exit or failure does not fail the run (e.g. optional setup). Default is false. |

Output start and end markers

When the command prints a lot of unwanted content (logs, progress, debug), use output.start and output.end in settings (or on the tool). The runner captures only the stdout between these two markers. Print the markers and the necessary output from your script.

JavaScript example: print the markers, then the result, then the end marker:

// In your script (e.g. getProjectDetails.js)
const startMarker = "__TOOL_OUTPUT_START__";
const endMarker = "__TOOL_OUTPUT_END__";

// ... do work, then:
console.log(startMarker);
console.log(JSON.stringify(result));
console.log(endMarker);

In tools/index.yaml you would set output.start and output.end to match (e.g. __TOOL_OUTPUT_START__ and __TOOL_OUTPUT_END__).

Supported arg types and optional args

Supported types: string, number, boolean, object. Use type in each arg.

Optional args: Add optional: true to an arg. If not provided, it may be omitted or passed empty depending on how you use it in exec (e.g. "${projectName}" for an optional string).

Example with multiple types and one optional arg:

tools:
  - name: "exampleTool"
    description: "Example with string, number, boolean, object args"
    args:
      - name: id
        type: string
      - name: count
        type: number
      - name: verbose
        type: boolean
      - name: options
        type: object
      - name: projectName
        type: string
        optional: true
    exec: node run.js ${id} ${count} ${verbose} "${options}" "${projectName}"

exec format

• Single line: exec: npm run getProjectDetails ${projectId}
• Multi-line: Use YAML multi-line (e.g. exec: | with following lines). Useful for running several commands in sequence (e.g. warmup, main command, cleanup):

exec: |
  npm run warmupTestingProcess
  npm run getProjectDetails ${projectId} "${projectName}"
  npm run cleanupTestingProcess

Full example (tools/index.yaml)

The following example shows settings (timeoutSec, output markers, env, ignore_failure), setup (one step with cwd and exec), and tools with optional arg and multi-line exec:

settings:
  timeoutSec: 10
  output:
    start: "__TOOL_OUTPUT_START__"
    end: "__TOOL_OUTPUT_END__"
  env:
    - BUILD_ENV: "dev"
    - DB_NAME: "legacy-db"
  ignore_failure: false
setup:
  - name: "Install dependencies"
    cwd: "api-tools"
    exec: "npm install"
tools:
  - name: "getProjectDetails"
    description: "Get project details from api"
    cwd: "api-tools"
    args:
      - name: projectId
        type: string
      - name: projectName
        type: string
        optional: true
    exec: |
      npm run warmupTestingProcess
      npm run getProjectDetails ${projectId} "${projectName}"
      npm run cleanupTestingProcess

Proxy

If the HTTP_PROXY or HTTPS_PROXY environment variables are set, the CLI will use them to proxy requests.

FAQ

How to add new test cases?
Add YAML files under .devassure/tests/ (or test/), or use a CSV file with --csv <path> and optionally .devassure/csv_mapping.yaml to map columns to test fields.

Is test case YAML always needed or can we use CSV only?
No. You can run tests from CSV only: use devassure run-tests --csv=<path> and configure .devassure/csv_mapping.yaml so your CSV columns map to summary, steps, priority, and tags.

What are the mandatory fields for a test case?
For YAML test cases, summary and steps are required. priority and tags are optional. For CSV, the mapped columns must provide at least the equivalent of summary and steps.

How to add a new environment?
Add a new top-level key (e.g. uat) and its data in .devassure/test_data.yaml. Then run with devassure run-tests --environment=uat (or --environment uat). If you omit --environment, the default key in test_data.yaml is used.

What are the supported browsers?
Supported browsers are Chromium, Chrome, Edge. Controlled by .devassure/preferences.yaml under browsers.default.browser. The default is chromium. See the preferences.yaml section above for the structure.

How to run with headless mode?
In .devassure/preferences.yaml, set browsers.default.headless: true.

How to run the tests in Continuous Integration?
Use devassure add-token <your-token> to authenticate (no browser). Then run devassure run-tests with the options you need (e.g. --csv, --filter, --tag, --priority, --archive for saving report zips). You can retain only recent sessions with devassure cleanup --retain-days 7 or --retain-sessions 10.

Support

For more information, visit DevAssure or run devassure help for command-specific help.