@openqa/cli

v3.1.2

Published

a month ago

Autonomous QA testing agent powered by Orka.js

0High
0Medium
0Low

qa testing automation ai orkajs autonomous agent e2e playwright test-generation

OpenQA is a truly autonomous QA testing agent that behaves like a senior QA engineer. It starts by understanding what kind of application it is facing — its domain, risk level, and regulatory requirements — then plans the testing strategy, seeds the Kanban board with strategic tasks, spawns specialist agents adapted to the project, and continuously improves based on findings.

✨ Features

🧠 Project Intelligence (Phase 0 — before the first test)

OpenQA reads the URL and optional codebase before running a single test, exactly like a QA lead who first studies the project:

Domain detection — fintech, healthcare, e-commerce, SaaS B2B, developer tools, government, media, education…
Risk level — critical, high, medium, low — drives testing depth
Regulatory context — automatically identifies applicable standards: PCI-DSS, GDPR, HIPAA, WCAG 2.1, SOC 2…
Mandatory checks — OWASP-referenced security checks selected for the detected domain
Critical paths — user flows that must work regardless (login, checkout, password reset…)
Testing strategy — exhaustive vs. standard depth, required test types, estimated session time

🗂️ Proactive Kanban

The moment intelligence runs, OpenQA populates the Kanban board automatically — without waiting for bugs to surface:

One ticket per mandatory security/compliance check
Improvement and tech-debt suggestions derived from the project profile
A QA strategy meta-ticket (domain, risk, critical paths, estimated time)
Live findings recorded as tickets during the session
Roadmap summary with prioritised counts (critical / high / medium / low)

🤖 Adaptive Specialist Agents

Specialists are selected by the intelligence layer, not hard-coded:

Pre-coded specialists: form-tester, security-scanner, sql-injection, xss-tester, component-tester, accessibility-tester, performance-tester, api-tester, auth-tester, navigation-tester
Dynamic agents (dynamic:<name>) — created on-the-fly from blueprints generated by the intelligence analysis. The brain invents agents it was never explicitly programmed with.

🧪 Test Generation

| Type | Description | |------|-------------| | Unit | Isolated function/component tests | | Functional | User workflow tests | | E2E | Complete user journey tests | | Security | SQL injection, XSS, auth bypass, CSRF… | | Regression | Bug-fix verification | | Performance | Load times, resource usage |

🔗 GitHub & GitLab Integration

OpenQA can audit any GitHub or GitLab repository without a running web app:

Code review — scans source files for security anti-patterns, missing tests, SQL concatenation, hardcoded secrets
Security audit — checks .env files, CI pipelines, auth code, CORS config
Issue analysis — identifies stale issues, recurring bug patterns, overloaded PRs
Dependency scan — flags known vulnerable packages (backend projects)
Auto Issues — creates GitHub issues for critical findings
GitLab support — works with gitlab.com and self-hosted GitLab instances

🔍 Human-in-the-Loop Approvals

Every finding gets a confidence score (0–100):

| Score | Action | |-------|--------| | ≥ 75 | Auto-approved — bug recorded immediately, persisted to DB | | 50–74 | Queued to /approvals for human review | | < 50 | Discarded |

The Approvals page lets reviewers approve or reject mid-confidence findings with one click. High/critical auto-approved findings are also automatically persisted as bug records for long-term tracking.

🕐 Scheduled QA Sessions

OpenQA includes a built-in cron scheduler — set recurring sessions without any external tooling:

# Accessible at http://localhost:4242/schedules
# Examples:
0 2 * * *       # Nightly at 2 AM
0 9 * * 1-5     # Weekdays at 9 AM
*/30 * * * *    # Every 30 minutes

The daemon checks every 60 seconds. Create, enable/disable, and delete schedules from the web UI or directly via the /api/schedules REST API.

📤 Export & Reports

Sessions can be exported in three formats:

GET /api/export/:sessionId?format=json — structured JSON (session + actions + bugs)
GET /api/export/:sessionId?format=csv — CSV with session summary, actions, and bugs sections
GET /api/export/:sessionId?format=html — styled HTML report with XSS-safe rendering

🐳 Docker Hub

OpenQA is available on Docker Hub at orkalab/openqa.

# Pull the latest image
docker pull orkalab/openqa:latest

# Run with your API key
docker run -d \
  -p 4242:4242 \
  -e LLM_PROVIDER=openai \
  -e OPENAI_API_KEY=sk-xxx \
  -e OPENQA_JWT_SECRET=$(openssl rand -hex 32) \
  -e SAAS_URL=https://your-app.com \
  -v openqa-data:/app/data \
  --name openqa \
  orkalab/openqa:latest

Then open http://localhost:4242 — first run will prompt you to create an admin account.

🚀 Quick Start

Development (Local)

# One-line installation
curl -fsSL https://openqa.orkajs.com/install.sh | bash

# Or via npm
npx @openqa/cli start

Production Deployment

# Interactive production installer
curl -fsSL https://openqa.orkajs.com/install-production.sh | bash

Supports:

🐳 Docker (recommended)
🖥️ VPS/Bare Metal (Ubuntu/Debian with systemd)
☁️ Cloud Platforms (Railway, Render, Fly.io)

📖 Configuration Guide: See CONFIGURATION.md for detailed configuration options
📖 Full Deployment Guide — Complete production setup instructions

Configure Your SaaS (3 lines!)

# Configure your application
curl -X POST http://localhost:4242/api/saas-config/quick \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My SaaS App",
    "description": "E-commerce platform with user auth, product catalog, and checkout",
    "url": "https://my-app.com"
  }'

# Start autonomous testing
curl -X POST http://localhost:4242/api/brain/run

That's it! OpenQA will:

Analyse the project — detect domain, risk, applicable regulations
Seed the Kanban — create strategic tickets before the first test
Select specialists — choose and create agents adapted to the project
Generate & execute tests — and report findings
Create GitHub issues for critical bugs

Manual Installation

git clone https://github.com/orka-js/openqa.git
cd openqa
npm install && npm run build
cp .env.example .env  # Add your LLM API key
npm start

📖 Usage

Start OpenQA

openqa start          # foreground (see logs)
openqa start --daemon # background

Access Web Interfaces

| Page | URL | Description | |---|---|---| | Dashboard | http://localhost:4242 | Live metrics, agents, activity | | Kanban | http://localhost:4242/kanban | Auto-generated QA tickets | | Sessions | http://localhost:4242/sessions | Session history | | Issues | http://localhost:4242/issues | Confirmed bugs | | Actions | http://localhost:4242/tests | Agent actions per session | | Coverage | http://localhost:4242/coverage | Pages and endpoints tested (raw counts) | | Approvals | http://localhost:4242/approvals | Mid-confidence findings for human review | | Schedules | http://localhost:4242/schedules | Cron-based autonomous QA schedules | | Logs | http://localhost:4242/logs | Full agent log stream | | Config | http://localhost:4242/config | Target and auth settings | | Environment | http://localhost:4242/config/env | LLM and server settings |

🔐 Dashboard Authentication

First-Time Setup

Visit http://localhost:4242
Create your admin username and password (min 8 characters)
You'll be automatically logged in

User Management (Admin Only)

# List all users
curl http://localhost:4242/api/accounts \
  -H "Authorization: Bearer YOUR_TOKEN"

# Create a viewer account
curl -X POST http://localhost:4242/api/accounts \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"username": "viewer1", "password": "securepass123", "role": "viewer"}'

# Change password
curl -X POST http://localhost:4242/api/auth/change-password \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"currentPassword": "old", "newPassword": "newsecure123"}'

Roles:

admin — Full access (manage users, configure, run tests)
viewer — Read-only access (view tests, bugs, sessions)

Security:

JWT-based authentication with httpOnly cookies
Scrypt password hashing
Rate limiting on auth endpoints
CSRF protection via SameSite cookies

Disable Authentication (Development Only)

export OPENQA_AUTH_DISABLED=true
openqa start

⚠️ Never disable authentication in production!

CLI Commands

openqa status               # Agent status
openqa logs                 # View logs
openqa logs --follow        # Stream logs
openqa config list          # All config values
openqa config get llm.provider
openqa config set saas.url https://your-app.com
openqa stop

⚙️ Configuration

Environment Variables

# LLM Configuration
LLM_PROVIDER=openai              # openai, anthropic, or ollama
OPENAI_API_KEY=sk-xxx
ANTHROPIC_API_KEY=
OLLAMA_BASE_URL=http://localhost:11434
LLM_MODEL=gpt-4                  # or claude-3-5-sonnet-20241022

# Target SaaS Application
SAAS_URL=https://your-app.com
SAAS_AUTH_TYPE=none              # none, basic, or session
SAAS_USERNAME=
SAAS_PASSWORD=

# GitHub Integration (Optional)
GITHUB_TOKEN=ghp_xxx
GITHUB_OWNER=your-username
GITHUB_REPO=your-repo

# Agent Behavior
AGENT_INTERVAL_MS=3600000        # 1 hour between test sessions
AGENT_MAX_ITERATIONS=20          # Max actions per session
AGENT_AUTO_START=true            # Start testing automatically

# Web UI
OPENQA_PORT=4242
OPENQA_HOST=0.0.0.0

# Database
DB_PATH=./data/openqa.db

See CONFIGURATION.md for the full reference.

🧠 How the Brain Works

Three Phases Per Session

┌─────────────────────────────────────────────────────────────────┐
│  PHASE 0 — PROJECT INTELLIGENCE                                 │
│  Detect domain · Assess risk · Identify regulations             │
│  Seed Kanban with strategic tasks                               │
└────────────────────────────┬────────────────────────────────────┘
                             ↓
┌─────────────────────────────────────────────────────────────────┐
│  PHASE 1 — LLM ANALYSIS                                         │
│  Deep understanding · Suggested tests · Risk areas              │
└────────────────────────────┬────────────────────────────────────┘
                             ↓
┌─────────────────────────────────────────────────────────────────┐
│  PHASE 2 — REACT LOOP (Think → Act → Observe)                   │
│  Specialist agents · Dynamic agents · Test generation           │
│  Bug recording · Kanban updates                                 │
└─────────────────────────────────────────────────────────────────┘

Phase 0 — Project Intelligence in detail

URL: https://payments.acme.com
         ↓
🔍 Heuristic signals: "pay", "stripe dep", "bcrypt dep", "multer dep"
         ↓
📊 Domain: fintech | Risk: CRITICAL | Regulations: PCI-DSS, GDPR
         ↓
📋 Mandatory checks: OWASP A02 (crypto), A03 (injection), A07 (auth)...
         ↓
🗂️  Kanban seeded: 12 tickets (7 security, 3 compliance, 2 perf)
         ↓
🤖 Specialists selected: auth-tester, xss-tester, sql-injection, api-tester
   + dynamic:PCI-DSS-Validator (blueprint invented on the fly)

A fintech app gets exhaustive security testing. A blog gets a standard profile. OpenQA adapts to the project, not the other way around.

Dynamic Agent Creation

🧠 Intelligence: "This fintech app needs a PCI-DSS specialist."
     ↓
🤖 Creates: dynamic:PCI-DSS-Validator
   - Role: PCI-DSS compliance tester
   - Goal: Verify cardholder data protection controls
   - Tools: browser, kanban (record findings)
     ↓
🧪 Runs: tests card data exposure, TLS versions, session timeouts
     ↓
🗂️  Kanban: tickets created for each finding

What It Generates

| Test Type | Example | |-----------|---------| | Unit | test_validateEmail() | | Functional | test_loginFlow() | | E2E | test_purchaseCheckout() | | Security | test_sqlInjection_loginForm() | | Regression | test_issue123_fix() | | Performance | test_checkoutPageLoad() |

📊 Web Interfaces

DevTools Dashboard

Current agent status and phase (Intelligence / Analysis / ReAct)
Live action logs
Test session history
Bugs detected with severity
Screenshots captured

Kanban Board

The board is seeded proactively with strategic tasks from the intelligence report, then updated live with findings:

| Column | Content | |--------|---------| | Backlog | Medium/low priority tasks, improvement ideas | | To Do | Critical/high security & compliance tasks, confirmed bugs | | In Progress | Being worked on | | Done | Resolved |

Each ticket has category tags (security, compliance, performance, improvement, tech-debt, missing-test) and OWASP references where applicable.

Configuration Panel

SaaS Info — Name, description, URL
Repository — Optional GitHub/GitLab URL for code analysis
Directives — Custom instructions guiding the agent
LLM Settings — Provider, API key, model
Git Listener — Auto-test on merge/deploy

🔧 Advanced Usage

Full Configuration

curl -X POST http://localhost:4242/api/saas-config \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My E-commerce App",
    "description": "Online store with Stripe checkout, user accounts, and product catalog",
    "url": "https://my-store.com",
    "repoUrl": "https://github.com/myorg/my-store",
    "directives": [
      "Focus on the checkout flow — it has had bugs before",
      "Test with both logged-in and guest users",
      "Check that discount codes work correctly"
    ],
    "auth": {
      "type": "session",
      "credentials": { "username": "[email protected]", "password": "testpass" }
    }
  }'

Add Directives On-the-fly

curl -X POST http://localhost:4242/api/saas-config/directive \
  -H "Content-Type: application/json" \
  -d '{"directive": "Pay special attention to the new refund feature"}'

Manage SaaS Config Directives

# List current directives
curl http://localhost:4242/api/saas-config

# Remove directive at index 0
curl -X DELETE http://localhost:4242/api/saas-config/directive/0

# Replace all directives
curl -X PUT http://localhost:4242/api/saas-config/directives \
  -H "Content-Type: application/json" \
  -d '{"directives": ["Check login", "Verify inbox", "Test compose"]}'

Connect Repository for Code Analysis

# The Brain will clone and analyse your code
curl -X POST http://localhost:4242/api/saas-config/repository \
  -H "Content-Type: application/json" \
  -d '{"url": "https://github.com/myorg/my-app"}'

Generate Specific Tests

# Generate a security test
curl -X POST http://localhost:4242/api/brain/generate-test \
  -H "Content-Type: application/json" \
  -d '{
    "type": "security",
    "target": "Login form SQL injection",
    "context": "The login form at /login accepts email and password"
  }'

Analyse Before Running

curl -X POST http://localhost:4242/api/brain/analyze
# Returns: understanding, suggestedTests, suggestedAgents, risks

Export a Session

# JSON export
curl "http://localhost:4242/api/export/SESSION_ID?format=json" > report.json

# CSV export
curl "http://localhost:4242/api/export/SESSION_ID?format=csv" > report.csv

# HTML report
curl "http://localhost:4242/api/export/SESSION_ID?format=html" > report.html

🚀 Production Deployment

Quick Deploy (5 minutes)

curl -fsSL https://openqa.orkajs.com/install-production.sh | bash

Deployment Options

| Method | Time | Difficulty | Best For | |--------|------|------------|----------| | 🐳 Docker | 5 min | Easy | VPS, Local servers | | 🖥️ VPS/Systemd | 15 min | Medium | Full control | | ☁️ Railway | 3 min | Easiest | Quick deploy | | 🎨 Render | 2 min | Easiest | Free tier | | 🪰 Fly.io | 5 min | Easy | Global edge |

Docker (Recommended)

git clone https://github.com/Orka-Community/OpenQA.git
cd OpenQA
cp .env.production .env
nano .env   # Required: OPENAI_API_KEY, OPENQA_JWT_SECRET, SAAS_URL
docker-compose -f docker-compose.production.yml up -d
# Open http://localhost:4242

With HTTPS (Nginx):

nano nginx.conf   # Update with your domain
sudo certbot certonly --standalone -d your-domain.com
docker-compose -f docker-compose.production.yml --profile with-nginx up -d

Cloud Platforms

Railway:

railway init && railway up
# Set env vars in dashboard: OPENAI_API_KEY, OPENQA_JWT_SECRET, SAAS_URL

Render: Fork repo → Connect to Render → Auto-deploys with render.yaml

Fly.io:

flyctl launch
flyctl secrets set OPENAI_API_KEY=sk-xxx OPENQA_JWT_SECRET=$(openssl rand -hex 32)
flyctl deploy

🔒 Security Checklist

[ ] Set strong OPENQA_JWT_SECRET (openssl rand -hex 32)
[ ] Use strong admin password (min 12 chars)
[ ] Enable HTTPS (SSL certificate)
[ ] Never set OPENQA_AUTH_DISABLED=true in production
[ ] Set NODE_ENV=production
[ ] Restrict CORS origins
[ ] Enable firewall (ports 80, 443 only)
[ ] Setup automated backups

📝 Architecture

openqa/
├── agent/
│   ├── intelligence/    # 🔍 Phase 0 — Project Intelligence Analyzer
│   │   └── index.ts         Domain, risk, regulations, mandatory checks, blueprints
│   ├── brain/           # 🧠 Phase 1-2 — Autonomous thinking & test generation
│   │   ├── index.ts         Main brain (3-phase runAutonomously loop)
│   │   ├── proactive-kanban.ts  Seed & update Kanban from intelligence + live findings
│   │   ├── diff-analyzer.ts     Git diff risk assessment
│   │   ├── llm-cache.ts         LLM response cache (TTL + LRU)
│   │   └── llm-resilience.ts    Retry, fallback, circuit-breaker
│   ├── specialists/     # 🤖 Pre-coded + dynamic specialist agents
│   ├── tools/           # Browser, GitHub, GitLab, Kanban tools
│   ├── config/          # SaaS configuration (ConfigManager, SaaSConfigManager)
│   ├── export/          # 📤 JSON, CSV, HTML export
│   └── webhooks/        # Git listener (GitHub/GitLab)
├── cli/                 # CLI commands & API routes
├── database/            # SQLite database
└── data/
    ├── workspace/        # Cloned repos
    └── generated-tests/  # Tests created by Brain

🔌 API Reference

SaaS Configuration

| Method | Path | Description | |--------|------|-------------| | GET | /api/saas-config | Get current SaaS config | | POST | /api/saas-config | Full configuration | | POST | /api/saas-config/quick | Quick setup (name, description, url) | | POST | /api/saas-config/directive | Add a directive | | DELETE | /api/saas-config/directive/:index | Remove directive by index | | PUT | /api/saas-config/directives | Replace all directives | | POST | /api/saas-config/repository | Set repo URL | | POST | /api/saas-config/local-path | Set local path |

Brain Control

| Method | Path | Description | |--------|------|-------------| | POST | /api/brain/analyze | Analyse app, get suggestions (Phase 1 only) | | POST | /api/brain/run | Start full autonomous session (Phase 0 → 1 → 2) | | POST | /api/brain/generate-test | Generate a specific test | | POST | /api/brain/create-agent | Create a custom dynamic agent | | POST | /api/brain/run-test/:id | Execute a generated test |

Data & Reports

| Method | Path | Description | |--------|------|-------------| | GET | /api/status | Agent status | | GET | /api/sessions | All sessions | | GET | /api/sessions/:id | Session detail | | GET | /api/sessions/:id/actions | Session actions | | GET | /api/bugs | All bugs found | | GET | /api/tests | Generated tests | | GET | /api/dynamic-agents | Created agents | | GET | /api/export/:sessionId?format=json\|csv\|html | Export session | | GET | /api/reports/:sessionId | Full report | | GET | /api/reports/:sessionId/download/html | Download HTML report | | GET | /api/reports/:sessionId/download/json | Download JSON report |

Kanban

| Method | Path | Description | |--------|------|-------------| | GET | /api/kanban/tickets | All tickets | | POST | /api/kanban | Create ticket | | PUT | /api/kanban/:id | Update ticket | | PATCH | /api/kanban/tickets/:id | Partial update (e.g. move column) | | DELETE | /api/kanban/:id | Delete ticket |

Approvals

| Method | Path | Description | |--------|------|-------------| | GET | /api/approvals | List findings (?status=pending\|approved\|rejected) | | PUT | /api/approvals/:id | Approve or reject a finding |

Schedules

| Method | Path | Description | |--------|------|-------------| | GET | /api/schedules | List all schedules | | POST | /api/schedules | Create a schedule | | PUT | /api/schedules/:id | Update / enable / disable | | DELETE | /api/schedules/:id | Delete a schedule |

Connection Testing

| Method | Path | Description | |--------|------|-------------| | POST | /api/test-connection | Test URL reachability + auth | | POST | /api/test-github | Test GitHub credentials | | POST | /api/test-llm | Test LLM provider |

WebSocket Events (`ws://localhost:4242`)

| Event | Description | |-------|-------------| | intelligence-complete | Phase 0 finished — domain, risk, regulations, checks | | kanban-seeded | Proactive Kanban tickets created (count + list) | | analysis-complete | Phase 1 LLM analysis finished | | thinking | Brain's current thought in the ReAct loop | | specialist-created | A specialist agent was instantiated | | specialist-completed | Specialist agent finished with findings | | test-generated | New test generated | | agent-created | New dynamic agent created | | test-started / test-completed | Test lifecycle events | | session-complete | Full session finished | | git-merge | Merge detected on main branch | | git-pipeline-success | CI/CD pipeline succeeded |

🛠️ Development

npm install
npm run dev    # Development mode (watch)
npm run build  # Production build
npm test       # Run test suite (vitest)

Test suite: 22 test files — 270 tests — covering database, API routes, export formats, config management, brain components (cache, resilience, diff analysis), auth, and full integration scenarios.

🤝 Contributing

Contributions welcome! Please read CONTRIBUTING.md first.

📄 License

MIT

🙏 Credits

Built with:

Orka.js — AI agent framework
Playwright — Browser automation
Next.js — Web interface
SQLite — Database

Made with ❤️ by the Orka Team