Cutlery Test Agent - Standalone Test Executor

A standalone, downloadable AI-powered test execution agent that developers and testers can run locally without needing the full Cutlery platform.

🎯 What is This?

The Cutlery Test Agent is a command-line tool that executes test cases defined in simple JSON files. It uses Playwright for browser automation and can run tests on any machine without requiring the full Cutlery web application.

📦 Installation

Download Pre-built Binary (Easiest)

macOS:

curl -L https://github.com/yourorg/cutlery-test-agent/releases/download/v1.0.0/cutlery-agent-macos -o cutlery-agent
chmod +x cutlery-agent
sudo mv cutlery-agent /usr/local/bin/

Linux:

curl -L https://github.com/yourorg/cutlery-test-agent/releases/download/v1.0.0/cutlery-agent-linux -o cutlery-agent
chmod +x cutlery-agent
sudo mv cutlery-agent /usr/local/bin/

Windows: Download cutlery-agent-win.exe from releases and add to PATH.

Install via npm

npm install -g @cutlery/test-agent

Install Playwright Browsers

After installation, install required browsers:

npx playwright install chromium
# Optional: install other browsers
npx playwright install firefox webkit

🚀 Quick Start

1. Create a Test Case

Create a file my-test.json:

{
  "name": "Login Test",
  "test_case_id": "TC_001",
  "automated_steps": [
    "Navigate to https://example.com/login",
    "Wait 1 second",
    "Fill \"[email protected]\" in \"email\"",
    "Fill \"password123\" in \"password\"",
    "Click \"Login\"",
    "Wait for \"Welcome\" to be visible",
    "Check if \"Dashboard\" is visible"
  ],
  "test_variables": {
    "email": "[email protected]",
    "password": "password123"
  }
}

2. Run the Test

cutlery-agent run my-test.json

3. See Browser in Action (Non-Headless)

cutlery-agent run my-test.json --no-headless

📋 Command Reference

Run a Test

cutlery-agent run <test-file.json> [options]

Options:

• -h, --headless - Run in headless mode (default: false)
• -b, --browser <type> - Browser type: chromium, firefox, webkit (default: chromium)
• -u, --base-url <url> - Base URL for the application
• -o, --output <path> - Output directory for screenshots/videos (default: ./test-results)
• -v, --verbose - Verbose output
• --openai-key <key> - OpenAI API key for AI features

Examples:

# Run with visible browser
cutlery-agent run test.json

# Run in headless mode
cutlery-agent run test.json --headless

# Use Firefox browser
cutlery-agent run test.json --browser firefox

# Custom output directory
cutlery-agent run test.json --output ./my-results

# Verbose output
cutlery-agent run test.json --verbose

Validate a Test Case

cutlery-agent validate <test-file.json>

Checks if your test case JSON is valid before running it.

Generate Example

cutlery-agent example [output-file.json]

Creates an example test case file you can modify.

📝 Test Case Format

Basic Structure

{
  "name": "Test Name",
  "test_case_id": "TC_ID",
  "automated_steps": ["step 1", "step 2"],
  "test_variables": {
    "variable1": "value1"
  }
}

Supported Step Types

Navigation

"Navigate to https://example.com"
"Go to https://example.com/page"

Waiting

"Wait 2 seconds"
"Wait for \"#button\" to be visible"
"Wait for \".loading\" to be visible"

Filling Forms

"Fill \"text\" in \"selector\""
"Fill \"[email protected]\" in \"input[name='email']\""
"Fill \"password\" in \"#password\""

Smart field matching (no selector needed):

"Fill \"[email protected]\" in \"email\""
"Fill \"password123\" in \"password\""

Clicking Elements

"Click \"button[type='submit'\""
"Click \"#login-button\""

Smart button matching (no selector needed):

"Click \"Login\""
"Click \"Submit\""
"Click \"Sign Up\""

Checking Visibility

"Check if \"Welcome\" is visible"
"Check if \"Error message\" is visible"

Screenshots

"Take screenshot"
"Capture screen"

Using Variables

Define variables and use them in steps:

{
  "test_variables": {
    "username": "testuser",
    "password": "secret123",
    "url": "https://example.com"
  },
  "automated_steps": [
    "Navigate to {{url}}",
    "Fill \"{{username}}\" in \"username\"",
    "Fill \"{{password}}\" in \"password\""
  ]
}

🔧 Configuration

Environment Variables

Create a .env file in your project:

# OpenAI API Key (optional, for AI features)
OPENAI_API_KEY=sk-...

# Default base URL
BASE_URL=http://localhost:3000

# Default browser
BROWSER_TYPE=chromium

# Headless mode
HEADLESS=true

OpenAI Integration (Optional)

For AI-enhanced test execution, provide an OpenAI API key:

export OPENAI_API_KEY=sk-...

Or use the flag:

cutlery-agent run test.json --openai-key sk-...

📊 Output

Console Output

📊 Test Case: Login Test
   Steps: 7
   Browser: chromium
   Headless: false
   Base URL: http://localhost:3000

✅ Test PASSED

📊 Test Results:
   Total Steps: 7
   Passed: 7
   Screenshots: 2

Screenshots

Screenshots are saved to the output directory:

• test-results/step-1-<timestamp>.png
• test-results/error-step-3-<timestamp>.png
• test-results/final-<timestamp>.png

🔄 CI/CD Integration

GitHub Actions

name: Run Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: "18"

      - name: Install Cutlery Agent
        run: npm install -g @cutlery/test-agent

      - name: Install Playwright browsers
        run: npx playwright install --with-deps chromium

      - name: Run tests
        run: cutlery-agent run tests/my-test.json --headless --verbose

      - name: Upload screenshots
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: test-screenshots
          path: test-results/

Jenkins

pipeline {
  agent any
  stages {
    stage('Setup') {
      steps {
        sh 'npm install -g @cutlery/test-agent'
        sh 'npx playwright install chromium'
      }
    }
    stage('Test') {
      steps {
        sh 'cutlery-agent run tests/*.json --headless --verbose'
      }
    }
  }
  post {
    always {
      archiveArtifacts artifacts: 'test-results/**/*', fingerprint: true
    }
  }
}

🎓 Examples

Example 1: Simple Login Test

{
  "name": "Login Flow",
  "automated_steps": [
    "Navigate to https://app.example.com",
    "Fill \"[email protected]\" in \"email\"",
    "Fill \"password123\" in \"password\"",
    "Click \"Sign In\"",
    "Check if \"Dashboard\" is visible"
  ]
}

Example 2: E-commerce Checkout

{
  "name": "Checkout Flow",
  "test_variables": {
    "product_url": "https://shop.example.com/products/123"
  },
  "automated_steps": [
    "Navigate to {{product_url}}",
    "Click \"Add to Cart\"",
    "Wait 1 second",
    "Click \"View Cart\"",
    "Click \"Proceed to Checkout\"",
    "Fill \"John Doe\" in \"input[name='name']\"",
    "Fill \"[email protected]\" in \"input[name='email']\"",
    "Click \"Place Order\"",
    "Check if \"Order Confirmed\" is visible"
  ]
}

Example 3: Form Validation

{
  "name": "Form Validation Test",
  "automated_steps": [
    "Navigate to https://example.com/signup",
    "Click \"Submit\"",
    "Check if \"Email is required\" is visible",
    "Fill \"invalid-email\" in \"email\"",
    "Click \"Submit\"",
    "Check if \"Invalid email format\" is visible",
    "Fill \"[email protected]\" in \"email\"",
    "Fill \"short\" in \"password\"",
    "Click \"Submit\"",
    "Check if \"Password must be at least 8 characters\" is visible"
  ]
}

🐛 Troubleshooting

Browsers Not Found

# Install browsers
npx playwright install chromium firefox webkit

Permission Denied (macOS/Linux)

chmod +x cutlery-agent

PATH Issues

Make sure /usr/local/bin is in your PATH:

echo $PATH
export PATH="/usr/local/bin:$PATH"

Test Fails with "Could not interpret step"

• Check step syntax matches examples above
• Use --verbose flag to see detailed error messages
• Try using explicit selectors instead of text matching

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Submit a pull request

📄 License

MIT License - see LICENSE file for details

🆘 Support

• GitHub Issues: https://github.com/yourorg/cutlery-test-agent/issues
• Documentation: https://docs.cutlery.dev
• Discord: https://discord.gg/cutlery

Made with ❤️ by the Cutlery Team