Supatest AI CLI

AI agent for E2E tests - Build, debug, and fix tests with AI assistance

Overview

Supatest AI is an AI-powered CLI that helps you write and maintain E2E tests. Use it interactively to build tests from scratch, or integrate it into CI to automatically fix failing tests.

Features

• Build Tests - Describe what to test in natural language, get working tests
• Debug Failures - AI investigates and fixes failing tests
• Fix in CI - Pipe test logs and let AI fix issues automatically

Requirements

• Node.js 18+ - Download Node.js
• Supatest Account - Sign up at supatest.ai

Getting Started

1. Install

npm install -g @supatest/cli

Or use npx without installing:

npx @supatest/cli

2. Login

Start the CLI and authenticate:

supatest

> /login

This opens your browser to authenticate with your Supatest account.

3. Setup

Check that your environment has the required tools:

> /setup

This verifies prerequisites like browsers and test frameworks are properly configured.

4. Discover

Scan your project to detect existing test files and configuration:

> /discover

This helps Supatest understand your test structure so it can work with your existing setup.

5. Start Testing

Now describe what you want to test:

> Write a test for the login flow
> Add a test that verifies users can add items to cart
> Test the checkout process with a valid credit card

Headless Mode (CI)

For CI/CD pipelines, set your API key and pass a task:

export SUPATEST_API_KEY="your-api-key"

# Fix failing tests from a log file
supatest "fix the failing tests" --logs test-output.log

# Pipe test output directly
npm test 2>&1 | supatest "fix any failures" --stdin

Options

supatest [task] [options]

Options:
  -l, --logs <file>       Path to log file to analyze
  --stdin                 Read logs from stdin
  --headless              Force headless mode (no interactive UI)
  -m, --max-iterations    Maximum AI turns (default: 100)
  -C, --cwd <path>        Working directory
  --verbose               Enable verbose logging
  -V, --version           Show version
  -h, --help              Show help

Commands:
  setup                   Check prerequisites and set up required tools

CI/CD Integration

GitHub Actions:

- name: Run tests
  id: tests
  continue-on-error: true
  run: npm test 2>&1 | tee test-output.log

- name: Fix failing tests
  if: steps.tests.outcome == 'failure'
  env:
    SUPATEST_API_KEY: ${{ secrets.SUPATEST_API_KEY }}
  run: npx @supatest/cli "fix the failing tests" --logs test-output.log

GitLab CI:

test:
  script:
    - npm test 2>&1 | tee test-output.log || true
    - npx @supatest/cli "fix failures" --logs test-output.log
  variables:
    SUPATEST_API_KEY: $SUPATEST_API_KEY

Advanced Features

LLM Provider Selection (/provider)

Supatest supports two LLM providers for running the AI agent:

Supatest Managed (Default)

Uses models available through the Supatest API. Your tasks are processed through Supatest infrastructure.

Claude Max

Uses your Claude Max subscription directly. This allows you to leverage Claude Max models while maintaining session tracking through Supatest.

Requirements for Claude Max:

• Active Claude Max subscription
• Claude Code login (automatic credential detection)
• Works on macOS, Linux, and Windows

Switching Providers:

In interactive mode, use the /provider command to switch between available providers:

> /provider

This opens a selector showing:

• Supatest Managed - Uses Supatest API with available models
• Claude Max - Direct Claude Max subscription (requires Claude Code login)

Navigate with arrow keys and press Enter to select. The current provider is highlighted.

Note: Provider selection is saved per session. You can also use Ctrl+P as a keyboard shortcut to cycle through providers.

Model Context Protocol (/mcp)

MCP (Model Context Protocol) servers extend Supatest with additional tools and capabilities. For example, the Playwright MCP server enables browser automation for test generation.

Viewing Configured Servers

> /mcp

Shows all configured MCP servers with:

• Server name and scope (project or global)
• Connection status (after testing)
• Command and arguments
• Environment variables (if any)
• Description (if provided)

Configuration Files

MCP servers are configured in JSON files. Supatest supports two levels:

Project-level (.supatest/mcp.json): Specific to your project, committed to version control.

Global (~/.supatest/mcp.json): Applies to all projects, stored in your home directory.

Project-level servers take precedence over global servers with the same name.

Configuration Format

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": ["@modelcontextprotocol/server-playwright"],
      "description": "Browser automation via Playwright",
      "enabled": true
    },
    "custom-tool": {
      "command": "./scripts/custom-mcp.js",
      "args": ["--option", "value"],
      "env": {
        "CUSTOM_VAR": "value"
      },
      "description": "Custom MCP server",
      "enabled": true
    }
  }
}

Fields:

• command (required) - Executable to run (e.g., npx, node, python)
• args (optional) - Arguments to pass to the command
• env (optional) - Environment variables to set for the server
• description (optional) - Human-readable description
• enabled (optional) - Whether the server is active (default: true)

Managing Servers Interactively

When viewing MCP servers with /mcp, you can:

• A - Add a new server (walks through configuration)
• R - Remove a selected server
• T - Test connection to all servers
• ESC or Q - Close the MCP view

Default Setup

When you run /setup, Supatest automatically configures the Playwright MCP server, which is essential for browser-based test generation and automation.

Example: Adding a Custom MCP Server

1. Create your MCP server (implements the Model Context Protocol)
2. Use /mcp and press A to add it
3. Or manually edit .supatest/mcp.json:

{
  "mcpServers": {
    "my-tool": {
      "command": "node",
      "args": ["/path/to/my-mcp-server.js"],
      "description": "My custom tool",
      "enabled": true
    }
  }
}

Debugging MCP Issues

If an MCP server isn't working:

1. View servers with /mcp
2. Press T to test all connections
3. Check error messages displayed for failed servers
4. Verify the command and arguments are correct
5. Ensure any required environment variables are set

Support

• Documentation
• GitHub Issues

License

ISC