ai-testing-suite

v1.3.0

Published

9 days ago

LangGraph Multi-Agent Automated Testing Suite for Node.js/TypeScript — 8 AI agents that analyze, test, review, secure & document your project. OWASP Top 10, Zero-Day scanning, 100+ vulnerability patterns.

AI Testing Suite

Multi-Agent LangGraph Orchestration for Automated Testing

8 specialized AI agents analyze your project, generate tests, review quality, execute tests, scan for vulnerabilities, and document everything — fully automated.

Installation · Quick Start · Agents · CLI Modes · Security · Reports

Overview

AI Testing Suite is a LangGraph-based multi-agent pipeline that automatically tests Node.js/TypeScript projects. It analyzes project structure and code, plans a test strategy, generates tailored test files, reviews their quality, executes them, scans for security vulnerabilities, and creates comprehensive reports.

Scanner → Analyzer → Strategist → Writer → Reviewer ──→ Runner → Security → Reporter
                                              │              ↑
                                              └── (Retry) ───┘

Why AI Testing Suite?

Extremely easy to integrate — A single command is all it takes to integrate the suite into any existing project
100% project-tailored — Tests are generated based on deep code analysis, not from templates
8 specialized agents — Each agent has a clearly defined role in the pipeline
OWASP Top 10 + Zero-Day — Comprehensive security scanning with 100+ vulnerability patterns
Console-driven — Interactive CLI menu with 8 modes
Production-readiness assessment — Clear verdict on whether the project is production-ready

Project Structure

ai-testing-suite/
├── src/
│   ├── index.ts                    # CLI Entry Point
│   ├── cli.ts                      # Interactive Console Menu
│   ├── config.ts                   # Configuration Management
│   ├── setup.ts                    # Quick Setup for Existing Projects
│   ├── types.ts                    # Complete TypeScript Type System
│   │
│   ├── agents/                     # 8 Specialized Agents
│   │   ├── scanner.agent.ts        # Agent 1: Project Structure Scanner
│   │   ├── analyzer.agent.ts       # Agent 2: Deep Code Analysis
│   │   ├── strategist.agent.ts     # Agent 3: Test Strategy Planner
│   │   ├── writer.agent.ts         # Agent 4: Test Generator
│   │   ├── reviewer.agent.ts       # Agent 5: Test Quality Reviewer
│   │   ├── runner.agent.ts         # Agent 6: Test Executor
│   │   ├── security.agent.ts       # Agent 7: Security Auditor
│   │   └── reporter.agent.ts       # Agent 8: Report Generator
│   │
│   ├── graph/                      # LangGraph Orchestration
│   │   ├── workflow.ts             # Pipeline Definition & Execution
│   │   ├── nodes.ts                # Graph Nodes & Conditional Edges
│   │   └── state.ts                # State Channel Management
│   │
│   └── utils/                      # Utility Functions
│       ├── parser.ts               # Code Parser (Functions, Classes, Patterns)
│       ├── file-utils.ts           # File System Operations
│       └── logger.ts               # Colored Console Output
│
├── tests/                          # Generated Tests (Output)
│   ├── unit/                       # Unit Tests
│   ├── integration/                # Integration Tests
│   ├── e2e/                        # End-to-End Tests
│   ├── security/                   # Security Tests
│   └── performance/                # Performance Tests
│
├── reports/                        # Generated Reports (Output)
│   ├── report-*.json               # Machine-readable
│   ├── report-*.md                 # Markdown Report
│   ├── report-*.html               # Visual HTML Report
│   └── agent-log-*.md              # Agent Activity Log
│
├── package.json
├── tsconfig.json
└── .env.example                    # Configuration Template

Installation

Install via npm (recommended)

npm install --save-dev ai-testing-suite

Then run the setup wizard:

npx ai-test-setup

This will:

Create test directories (tests/unit, tests/integration, tests/e2e, tests/security, tests/performance)
Install the right dependencies (Vitest or Jest, Playwright, etc.)
Add npm scripts to your package.json
Create .env.example with configuration template

Quick Start in 3 Steps

# 1. Install
npm install --save-dev ai-testing-suite

# 2. Configure
cp .env.example .env
# Enter your API key (OpenAI or Anthropic) or use Ollama locally

# 3. Run
npx ai-test --mode full

Setup Flags

npx ai-test-setup              # Default: Vitest + Playwright
npx ai-test-setup --jest        # Use Jest instead of Vitest
npx ai-test-setup --no-playwright  # Skip Playwright installation

Alternative: Clone from GitHub

git clone https://github.com/zurd46/AI-Testing-Suite.git
cd AI-Testing-Suite
npm install
npm run setup -- /path/to/project

Quick Start

Interactive Mode (recommended)

npm start
# or
npx ts-node src/index.ts

Displays the interactive menu:

╔══════════════════════════════════════════════════════════════╗
║  AI TESTING SUITE                                          ║
║  Multi-Agent LangGraph Orchestration                        ║
╠══════════════════════════════════════════════════════════════╣
║  8 Agents | OWASP Top 10 | Zero-Day Scans | 100% Coverage  ║
╚══════════════════════════════════════════════════════════════╝

Select a mode:

  1) Full Pipeline     - Complete analysis, tests & security
  2) Analyze Only      - Project structure & code analysis
  3) Generate Tests    - Analysis + create & review tests
  4) Run Tests         - Execute existing tests
  5) Security Scan     - Security audit only
  6) Custom Pipeline   - Choose your own agent combination
  7) Generate Report   - Create report
  8) Configuration     - View/modify settings
  0) Exit

Direct Modes

# Full pipeline
npm run full -- --path /path/to/project

# Analysis only
npm run analyze -- --path /path/to/project

# Generate tests only
npm run generate -- --path /path/to/project

# Security scan only
npm run security -- --path /path/to/project

# Report only
npm run report -- --path /path/to/project

CLI Options

npx ts-node src/index.ts \
  --mode full \
  --path /path/to/project \
  --output ./tests \
  --reports ./reports \
  --provider openai \
  --model gpt-4o \
  --depth deep \
  --verbose

| Option | Description | Default | |--------|-------------|---------| | -m, --mode | Mode: full\|analyze\|generate\|run\|security\|report\|interactive | interactive | | -p, --path | Path to target project | ./ | | -o, --output | Test output directory | ./tests | | -r, --reports | Report directory | ./reports | | --provider | LLM Provider: openai\|anthropic\|openai_compatible\|ollama | openai | | --model | LLM Model | gpt-4o | | --runner | Test runner: vitest\|jest\|node | vitest (auto-detected) | | --e2e | E2E runner: playwright\|supertest\|none | playwright (auto-detected) | | --coverage | Coverage tool: v8\|c8\|istanbul | v8 (auto-detected) | | --depth | Security scan depth: basic\|standard\|deep | deep | | -v, --verbose | Verbose output | true | | --no-security | Disable security scanning | - | | --no-zero-day | Disable zero-day checks | - |

The 8 Agents

Agent 1: Scanner

Scans the complete project structure

Detects framework (Next.js, NestJS, Express, React, Vue, Angular, ...)
Identifies entry points, config files, package manager
Counts files, lines, languages
Detects features (TypeScript, Prisma, GraphQL, WebSockets, Redis, ...)

Agent 2: Analyzer

Deep code analysis of every source file

Parses functions, classes, interfaces, variables
Detects API endpoints (Express, NestJS Decorators)
Detects database operations (Query, Insert, Update, Delete)
Builds dependency graph with circular dependency detection
Detects design patterns (Singleton, Factory, Observer, Middleware, ...)
Calculates cyclomatic complexity
Detects error handling patterns

Agent 3: Strategist

Plans the optimal test strategy

Creates unit test plans per function/class
Plans API integration tests per endpoint
Plans database integration tests
Plans E2E tests per resource
Plans security tests (OWASP-based)
Plans performance tests
Prioritizes by criticality and complexity

Generates test cases for each category: happy-path · edge-case · error-handling · boundary · null-undefined · type-safety · concurrency · security · performance · regression

Agent 4: Writer

Generates tailored test files

Writes unit tests with correct imports and mocks
Writes integration tests with Supertest
Writes E2E tests with Playwright (browser) or Supertest (API)
Writes security tests with real attack payloads
Writes performance tests (latency, concurrency, memory)
Generates runner-specific config (vitest.config.ts, jest.config.ts, playwright.config.ts)
Auto-adapts syntax per runner (vi.fn() vs jest.fn(), imports vs globals)
Creates directory structure automatically

Agent 5: Reviewer

Reviews generated tests for quality

Checks test structure (describe, it, beforeEach)
Counts and evaluates assertions
Checks test isolation (mock reset)
Checks edge case coverage
Checks error handling tests
Checks mock usage and spy assertions
Checks async patterns
Detects placeholder assertions
Calculates quality score (0-100)
Conditional Edge: If score < threshold, pipeline loops back to the Writer

Agent 6: Runner

Executes all tests and collects results

Supports Vitest (default, fastest), Jest, and Node.js built-in test runner
Runs Playwright for E2E browser tests
Runs tests grouped by type (Unit, Integration, E2E, Security, Performance)
Parses JSON output from each runner
Collects errors with test name, message, stack trace
Auto-detects installed test runner from target project
Dry run when runner is not installed
Calculates success rate

Agent 7: Security Auditor

Comprehensive security audit

Vulnerability Checks (100+ Patterns):

| Category | Checks | |----------|--------| | SQL Injection | Template Literals, String Concatenation, Raw Queries | | XSS | innerHTML, dangerouslySetInnerHTML, document.write, Reflected Input | | Command Injection | exec(), execSync(), spawn() with user input | | Path Traversal | readFile/writeFile with user input, path.join manipulation | | Prototype Pollution | Object.assign, Spread, proto, constructor | | SSRF | fetch/axios/got with user-controlled URL | | CSRF | State-changing endpoints without token | | ReDoS | User input in RegExp, nested quantifiers | | Hardcoded Secrets | Passwords, API keys, private keys, connection strings | | Insecure Randomness | Math.random() for tokens/keys | | Broken Auth | JWT "none" algorithm, weak hashes, TLS bypass | | Information Disclosure | Stack traces in responses, sensitive data in logs | | Zero-Day Patterns | process.binding, dynamic Function(), Proxy manipulation, Buffer.allocUnsafe |

Additionally:

Dependency vulnerability audit (known CVEs in npm packages)
OWASP Top 10 compliance check (A01-A10:2021)
Code smell detection (TODO, empty catch blocks, console.log in production)
Risk score calculation (0-100)

Agent 8: Reporter

Creates comprehensive documentation

JSON Report — Machine-readable, for CI/CD integration
Markdown Report — Human-readable, for code reviews
HTML Report — Visual report with charts and colors
Agent Log — Chronological log of all agent actions
Production-readiness assessment — YES/NO with reasoning
Recommendations — Concrete steps for improvement

CLI Modes

| Mode | Description | Agents | |------|-------------|--------| | full | Complete pipeline | All 8 | | analyze | Analysis only | Scanner → Analyzer | | generate | Generate tests | Scanner → Analyzer → Strategist → Writer → Reviewer | | run | Run tests | Scanner → Analyzer → Strategist → Writer → Runner | | security | Security scan | Scanner → Analyzer → Security → Reporter | | report | Generate report | Reporter | | interactive | Interactive menu | Selectable | | Custom | Custom combination | Freely selectable (e.g. 1,2,7) |

Supported Test Runners

AI Testing Suite auto-detects the test runner from your project. You can also override via CLI or .env.

| Runner | Type | Speed | Best For | |--------|------|-------|----------| | Vitest (default) | Unit/Integration | Fastest | Modern TypeScript projects, ESM | | Jest | Unit/Integration | Fast | Existing Jest projects, CJS | | Node.js test runner | Unit | Fast | Zero-dependency, Node >= 18 | | Playwright (default) | E2E/Browser | Medium | Frontend/Fullstack, cross-browser | | Supertest | E2E/API | Fast | Backend API testing |

Coverage Tools

| Tool | Speed | Integration | |------|-------|-------------| | v8 (default) | Fastest | Native V8 engine coverage | | c8 | Fast | V8 coverage via CLI wrapper | | istanbul | Standard | Traditional instrumentation |

Auto-Detection

The Scanner agent automatically detects your project's existing test tooling:

vitest in package.json    → uses Vitest
jest/ts-jest              → uses Jest
@playwright/test          → uses Playwright for E2E
supertest                 → uses Supertest for E2E
c8                        → uses c8 for coverage
nyc/istanbul              → uses Istanbul for coverage

Override with CLI: npx ai-test --mode full --runner jest --e2e playwright --coverage v8

Security Scanning

OWASP Top 10 (2021) Coverage

| # | Category | Status | |---|----------|--------| | A01 | Broken Access Control | Covered | | A02 | Cryptographic Failures | Covered | | A03 | Injection | Covered | | A04 | Insecure Design | Covered | | A05 | Security Misconfiguration | Covered | | A06 | Vulnerable Components | Covered | | A07 | Auth Failures | Covered | | A08 | Software Integrity | Covered | | A09 | Logging Failures | Covered | | A10 | SSRF | Covered |

Zero-Day Pattern Detection

Detection of patterns that may indicate potential zero-day exploits:

process.binding() — Access to internal Node.js APIs
Function() with user input — Dynamic code execution
Proxy with user input — Handler manipulation
Reflect API with user input — Metaprogramming attacks
Dynamic import() with user input — Module injection
constructor bracket notation — Sandbox escape
Buffer.allocUnsafe() — Uninitialized memory (memory leak)
globalThis/global[] — Global scope manipulation

Reports

HTML Report

The HTML report provides a visual overview with:

Overall status (Production-ready YES/NO)
Score cards (Tests, Security, Quality)
Vulnerability table with severity colors
OWASP compliance status
Recommendations

Markdown Report

Contains:

Summary with metrics
Project structure details
Code analysis results
Generated tests by type
Test results with errors
Security report with all vulnerabilities
OWASP status
Recommendations

JSON Report

Complete, machine-readable report for CI/CD integration:

{
  "projectName": "my-app",
  "timestamp": "2025-01-15T10:30:00Z",
  "summary": {
    "totalTests": 142,
    "totalPassed": 138,
    "totalFailed": 4,
    "securityScore": 85,
    "qualityScore": 92,
    "productionReady": false,
    "recommendations": [...]
  },
  "security": {
    "vulnerabilities": [...],
    "owaspChecks": [...],
    "overallRisk": "medium"
  }
}

Configuration

Environment Variables (.env)

# LLM Provider
OPENAI_API_KEY=sk-...           # OpenAI API Key
ANTHROPIC_API_KEY=sk-ant-...    # Or Anthropic API Key
LLM_PROVIDER=openai             # openai | anthropic | openai_compatible | ollama
LLM_MODEL=gpt-4o               # Model name (ollama default: llama3.2)
LLM_TEMPERATURE=0.1            # Low for consistent tests
LLM_MAX_TOKENS=4096

# Paths
TARGET_PROJECT_PATH=./          # Target project
TESTS_OUTPUT_DIR=./tests        # Test output
REPORTS_OUTPUT_DIR=./reports    # Report output

# Test Runner Configuration
TEST_RUNNER=vitest              # vitest (default, fastest) | jest | node
E2E_RUNNER=playwright           # playwright (default) | supertest | none
COVERAGE_TOOL=v8                # v8 (default, native) | c8 | istanbul

# Security
SECURITY_SCAN_DEPTH=deep       # basic | standard | deep
CHECK_ZERO_DAY=true
CHECK_OWASP_TOP_10=true
CHECK_DEPENDENCIES=true

# Agent
MAX_AGENT_ITERATIONS=10
AGENT_VERBOSE=true

Supported Frameworks

| Framework | Type | Detection | |-----------|------|-----------| | Next.js | Fullstack | Automatic | | NestJS | Backend | Automatic | | Express | Backend | Automatic | | Fastify | Backend | Automatic | | Koa | Backend | Automatic | | Hapi | Backend | Automatic | | React | Frontend | Automatic | | Vue | Frontend | Automatic | | Angular | Frontend | Automatic | | Svelte | Frontend | Automatic | | Nuxt | Fullstack | Automatic | | Remix | Fullstack | Automatic | | Electron | Fullstack | Automatic | | Node.js | Library/CLI | Fallback |

Detected Features

TypeScript · Prisma · TypeORM · Mongoose · Sequelize · GraphQL · WebSockets · Redis · Passport · JWT · bcrypt · Helmet · CORS · Joi/Zod/Yup · Winston/Pino · Bull/BullMQ · Swagger/OpenAPI · Vitest · Playwright · Stryker · c8 · ESLint Security · PactumJS

Prerequisites

Node.js >= 18.0.0
npm, yarn, or pnpm
TypeScript (recommended, JavaScript is also supported)
API Key for OpenAI or Anthropic (for LLM-powered features), or Ollama for local inference (no API key needed)

Example Output

======================================================================
  AI TESTING SUITE - FULL MODE
======================================================================

-- Project: /home/user/my-app
-- Pipeline: scanner -> analyzer -> strategist -> writer -> reviewer -> runner -> security -> reporter
-- 8 agents will be executed

>> [SCAN] Agent started...
   [SCAN] 47 files found
   [SCAN] 12 directories found
   [SCAN] Framework detected: Express (backend)
OK [SCAN] Agent completed (0.3s)

>> [ANALYZE] Agent started...
   [ANALYZE] 23 modules analyzed
   [ANALYZE] 45 functions, 8 classes
   [ANALYZE] 12 API endpoints found
OK [ANALYZE] Agent completed (1.2s)

>> [STRATEGY] Agent started...
   Test Type       | Plans | Test Cases
   Unit Tests      |    18 |         72
   Integration     |    14 |         42
   E2E Tests       |     4 |          8
   Security Tests  |     2 |         16
   Performance     |     1 |          3
   TOTAL           |    39 |        141
OK [STRATEGY] Agent completed (0.8s)

>> [WRITE] Agent started...
   [WRITE] 18 unit test files written
   [WRITE] 14 integration test files written
   [WRITE] 4 E2E test files written
OK [WRITE] Agent completed (2.1s)

>> [REVIEW] Agent started...
   36/39 passed (Score: 87.3)
OK [REVIEW] Agent completed (0.5s)

>> [RUN] Agent started...
   PASS auth.test.ts (124ms)
   PASS users.test.ts (89ms)
   FAIL payment.test.ts (203ms)
OK [RUN] Agent completed (12.4s)

>> [SECURITY] Agent started...
   [CRITICAL] hardcoded-credentials: Hardcoded API key (config.ts)
   [HIGH]     sql-injection: Raw SQL with Concatenation (users.service.ts)
   [MEDIUM]   xss: innerHTML assignment (dashboard.tsx)
   Security Score: 72/100
OK [SECURITY] Agent completed (1.8s)

>> [REPORT] Agent started...
   Reports saved to: ./reports
OK [REPORT] Agent completed (0.4s)

======================================================================
  TEST RESULTS
======================================================================
  Project:            my-app
  Framework:          Express (backend)
  Total Tests:        141
  Passed:             137
  Failed:             4
  Security Score:     72/100
  Quality Score:      87/100

!! NOT PRODUCTION-READY - Please review recommendations:
??   -> Fix 4 failing tests
??   -> Address security vulnerabilities
??   -> Fix 1 critical vulnerability immediately!

License

MIT

Built with LangGraph, TypeScript, and the power of 8 AI agents.