runhuman

AI-orchestrated human QA testing from your terminal

What is Runhuman?

Runhuman is an AI-orchestrated human QA testing service. Get real human testing of your web applications on-demand, with structured results extracted automatically.

Perfect for:

• AI coding agents that need human verification
• CI/CD pipelines with human QA gates
• Visual/UX testing requiring real human judgment
• On-demand QA without hiring/managing teams

Installation

# Install globally
npm install -g runhuman

# Or use with npx
npx runhuman --help

Quick Start

# 1. Login with your API key
runhuman login

# 2. Create your first test
runhuman create https://myapp.com -d "Test the checkout flow"

# 3. Check the status
runhuman status <jobId>

# 4. Get results
runhuman results <jobId>

Commands

Jobs

create [url]

Create a new QA test job.

# Basic usage
runhuman create https://myapp.com -d "Test checkout flow"

# With template
runhuman create https://myapp.com --template tmpl_abc123

# Synchronous (wait for result)
runhuman create https://myapp.com -d "Test" --sync

# With custom schema
runhuman create https://myapp.com -d "Test" --schema ./schema.json

Options:

• -d, --description <text> - Test instructions for the human tester
• --template <id> - Use a pre-defined template
• --duration <seconds> - Target test duration
• --screen-size <size> - Screen size (desktop/mobile/tablet)
• --schema <path> - Path to JSON schema for structured output
• --sync - Wait for result before exiting
• --json - Output as JSON

status <jobId>

Check the status of a test job.

runhuman status job_abc123

wait <jobId>

Wait for a job to complete and display results.

runhuman wait job_abc123
runhuman wait job_abc123 --timeout 300

Options:

• --timeout <seconds> - Maximum wait time (default: 600)

results <jobId>

Display detailed test results.

runhuman results job_abc123
runhuman results job_abc123 --json

list

List all your test jobs.

# List all jobs
runhuman list

# Filter by status
runhuman list --status completed

# Filter by project
runhuman list --project proj_abc123

# Limit results
runhuman list --limit 20

Options:

• --status <status> - Filter by status (pending/claimed/in_progress/completed/failed)
• --project <id> - Filter by project
• --limit <number> - Maximum number of jobs to return
• --json - Output as JSON

Projects

projects list

List all your projects.

runhuman projects list
runhuman projects ls  # alias

projects create <name>

Create a new project.

runhuman projects create "My App"
runhuman projects create "My App" -d "Production QA testing"

Options:

• -d, --description <text> - Project description
• --default-url <url> - Default URL to test
• --github-repo <owner/repo> - Link GitHub repository

projects show <projectId>

Show details of a specific project.

runhuman projects show proj_abc123
runhuman projects info proj_abc123  # alias

projects update <projectId>

Update a project.

runhuman projects update proj_abc123 --name "New Name"
runhuman projects update proj_abc123 --description "New description"

Options:

• --name <name> - New project name
• -d, --description <text> - New description
• --default-url <url> - New default URL
• --github-repo <owner/repo> - New GitHub repository

projects delete <projectId>

Delete a project permanently.

runhuman projects delete proj_abc123 --force

Options:

• --force - Skip confirmation prompt

API Keys

keys list

List all API keys for a project.

runhuman keys list --project proj_abc123
runhuman keys ls --project proj_abc123  # alias

Options:

• --project <id> - Project ID (required)
• --show-keys - Show full API keys (default: masked)

keys create <name>

Create a new API key.

runhuman keys create "CI/CD Key" --project proj_abc123
runhuman keys new "CI/CD Key" --project proj_abc123  # alias

Options:

• --project <id> - Project ID (required)
• --copy - Copy key to clipboard

keys delete <keyId>

Delete an API key permanently.

runhuman keys delete key_abc123 --force
runhuman keys rm key_abc123 --force  # alias

Options:

• --force - Skip confirmation prompt

Templates

templates list

List all test templates for a project.

runhuman templates list --project proj_abc123
runhuman templates ls --project proj_abc123  # alias

Options:

• --project <id> - Project ID (required)

templates create <name>

Create a new test template.

runhuman templates create "Checkout Flow" --project proj_abc123
runhuman templates create "Login Test" --project proj_abc123 \
  -d "Test login functionality" \
  --duration 180 \
  --screen-size mobile \
  --schema ./schema.json

Options:

• --project <id> - Project ID (required)
• -d, --description <text> - Template description
• --duration <seconds> - Target test duration
• --screen-size <size> - Default screen size
• --schema <path> - Path to JSON schema file

templates show <templateId>

Show details of a specific template.

runhuman templates show tmpl_abc123
runhuman templates info tmpl_abc123  # alias

templates update <templateId>

Update a template.

runhuman templates update tmpl_abc123 --name "New Name"
runhuman templates update tmpl_abc123 --description "New description"
runhuman templates update tmpl_abc123 --schema ./new-schema.json

Options:

• --name <name> - New template name
• -d, --description <text> - New description
• --duration <seconds> - New target duration
• --screen-size <size> - New default screen size
• --schema <path> - Path to new JSON schema file

templates delete <templateId>

Delete a template permanently.

runhuman templates delete tmpl_abc123 --force
runhuman templates rm tmpl_abc123 --force  # alias

Options:

• --force - Skip confirmation prompt

GitHub Integration

github link <repo>

Link a GitHub repository to your project.

runhuman github link owner/repo --project proj_abc123
runhuman gh link volter-ai/runhuman --project proj_abc123  # alias

Options:

• --project <id> - Project ID (required)

github repos

List linked GitHub repositories.

runhuman github repos
runhuman github repos --project proj_abc123  # filter by project

Options:

• --project <id> - Filter by project

github issues <repo>

List GitHub issues for a repository.

runhuman github issues owner/repo
runhuman github issues owner/repo --state open
runhuman github issues owner/repo --labels "bug,needs-qa"

Options:

• --state <state> - Filter by state (open/closed/all, default: open)
• --labels <labels> - Filter by comma-separated labels

github test <issueNumber>

Create a QA test job for a GitHub issue.

runhuman github test 123 --repo owner/repo --url https://myapp.com
runhuman github test 123 --url https://myapp.com --sync  # wait for result

Options:

• --repo <owner/repo> - Repository (or use default from config)
• --url <url> - URL to test (required)
• --template <id> - Template ID to use
• --sync - Wait for result before exiting

github bulk-test

Create QA test jobs for multiple GitHub issues.

# Test all open issues
runhuman github bulk-test --repo owner/repo --url https://myapp.com

# Test issues with specific labels
runhuman github bulk-test --repo owner/repo --url https://myapp.com \
  --labels "bug,needs-qa" \
  --limit 5

Options:

• --repo <owner/repo> - Repository (or use default from config)
• --url <url> - URL to test (required)
• --labels <labels> - Filter issues by comma-separated labels
• --state <state> - Filter by state (open/closed/all, default: open)
• --template <id> - Template ID to use
• --limit <number> - Maximum number of jobs to create (default: 10)

Authentication

login

Login to Runhuman and save your API key.

runhuman login
# Prompts for API key

# Or provide directly
runhuman login --api-key qa_live_xxxxxxxxxxxxx

Get your API key at: runhuman.com/dashboard

logout

Logout and clear saved credentials.

runhuman logout

whoami

Display current user info and account balance.

runhuman whoami

Configuration

config get <key>

Get a configuration value.

runhuman config get apiUrl
runhuman config get project

config set <key> <value>

Set a configuration value.

# Set globally
runhuman config set apiUrl https://api.runhuman.com --global

# Set for current project
runhuman config set project proj_abc123
runhuman config set defaultUrl https://myapp.com

Options:

• --global - Save to global config instead of project config

config list

List all configuration values.

runhuman config list

Shows configuration from all sources:

• Global config (~/.config/runhuman/config.json)
• Project config (.runhumanrc)
• Environment variables
• Effective (merged) config

config reset

Reset configuration to defaults.

# Reset project config
runhuman config reset project

# Reset global config
runhuman config reset global

# Reset all
runhuman config reset all

init

Initialize a new Runhuman project with interactive prompts.

runhuman init

Creates a .runhumanrc file in the current directory with your project configuration.

Configuration

Configuration Hierarchy

Configuration is loaded from multiple sources with the following priority (highest to lowest):

1. CLI flags - --api-key, --api-url, etc.
2. Environment variables - RUNHUMAN_API_KEY, RUNHUMAN_API_URL, etc.
3. Project config - .runhumanrc in current directory
4. Global config - ~/.config/runhuman/config.json
5. Defaults

Environment Variables

RUNHUMAN_API_KEY         # API key for authentication
RUNHUMAN_API_URL         # API base URL (default: https://runhuman.com)
RUNHUMAN_PROJECT         # Default project ID
RUNHUMAN_DEFAULT_URL     # Default URL to test
RUNHUMAN_DEFAULT_DURATION  # Default test duration in seconds
RUNHUMAN_DEFAULT_SCREEN_SIZE  # Default screen size (desktop/mobile/tablet)
RUNHUMAN_OUTPUT_FORMAT   # Output format (pretty/json/compact)
RUNHUMAN_NO_COLOR=1      # Disable colored output

Project Config (.runhumanrc)

Create a .runhumanrc file in your project root:

{
  "project": "proj_abc123",
  "defaultUrl": "https://myapp.com",
  "defaultDuration": 300,
  "defaultScreenSize": "desktop",
  "githubRepo": "owner/repo",
  "outputFormat": "pretty",
  "color": true
}

Global Config

Stored at ~/.config/runhuman/config.json:

{
  "apiUrl": "https://runhuman.com",
  "apiKey": "qa_live_xxxxxxxxxxxxx",
  "color": true
}

JSON Output

All commands support --json flag for machine-readable output:

runhuman create https://myapp.com -d "Test" --json
runhuman status job_abc123 --json
runhuman list --json

JSON output format:

{
  "success": true,
  "data": { ... },
  "error": null
}

Exit Codes

• 0 - Success
• 1 - General error
• 2 - Authentication error
• 3 - Not found error
• 4 - Validation error
• 5 - Timeout error

Examples

Basic Workflow

# 1. Login
runhuman login

# 2. Initialize project
runhuman init

# 3. Create a test
runhuman create https://myapp.com -d "Test the login flow"

# 4. Wait for results
runhuman wait job_abc123

# 5. View detailed results
runhuman results job_abc123

CI/CD Integration

# Create test and wait for result (fail on error)
runhuman create https://staging.myapp.com \
  -d "Smoke test on staging" \
  --sync \
  --json > result.json

# Exit code will be non-zero if test fails

GitHub Integration Workflow

# 1. Link your repo
runhuman github link owner/repo --project proj_abc123

# 2. List issues needing QA
runhuman github issues owner/repo --labels "needs-qa"

# 3. Test a specific issue
runhuman github test 42 --url https://myapp.com --sync

# 4. Bulk test multiple issues
runhuman github bulk-test --url https://myapp.com --labels "needs-qa" --limit 5

Using Templates

# 1. Create a template
runhuman templates create "Login Flow Test" --project proj_abc123 \
  -d "Test login with valid and invalid credentials" \
  --duration 180 \
  --schema ./schemas/login-test.json

# 2. Use the template
runhuman create https://myapp.com --template tmpl_abc123

# 3. List all templates
runhuman templates list --project proj_abc123

MCP Server for AI Agents

If you're using AI agents (Claude, GPT, etc.), check out our MCP server package:

npm install -g @runhuman/mcp

See @runhuman/mcp for details.

Learn More

• Website: runhuman.com
• Documentation: runhuman.com/docs
• GitHub: github.com/volter-ai/runhuman

Support

• Email: [email protected]
• Issues: GitHub Issues

How It Works

1. You create a job - Define test instructions and optional output schema
2. Job posted to testers - Request goes to human tester pool
3. Human performs test - Real person tests with video/screenshot recording
4. AI extracts data - GPT-4o processes feedback into structured JSON
5. You get results - Clean, typed data ready for automation

Pricing: Pay-per-second ($0.0018/sec of tester time). No monthly fees, no minimums.

Built with ❤️ by the Volter AI team