systemeval-ts

TypeScript test evaluation framework for integration and E2E testing. A TypeScript port of the Python systemeval package with Jest and Playwright adapter support.

Homepage: debugg.ai | Docs: debugg.ai/docs/systemeval

Installation

npm install systemeval-ts

Or globally for CLI usage:

npm install -g systemeval-ts

Quick Start

CLI Usage

# Initialize configuration
systemeval-ts init

# Run tests with Jest
systemeval-ts test

# Run tests with Playwright
systemeval-ts test --browser

# Discover available tests
systemeval-ts discover

# List available adapters
systemeval-ts list adapters

Programmatic Usage

import {
  createAdapter,
  createJestAdapter,
  createPlaywrightAdapter,
  Verdict,
} from 'systemeval-ts';

// Using the adapter registry
const adapter = createAdapter('jest', '/path/to/project');
const result = await adapter.execute({
  parallel: true,
  failfast: false,
});

console.log(`Verdict: ${result.verdict}`);
console.log(`Passed: ${result.passed}, Failed: ${result.failed}`);

// Direct adapter instantiation
const jestAdapter = createJestAdapter('/path/to/project', {
  configFile: 'jest.config.js',
  coverage: true,
});

const playwrightAdapter = createPlaywrightAdapter('/path/to/project', {
  configFile: 'playwright.config.ts',
  headed: false,
  browser: 'chromium',
});

Configuration

Create a systemeval.yaml file in your project root:

# Default adapter to use
adapter: jest

# Project root directory
projectRoot: .

# Test categories
categories:
  unit:
    patterns:
      - "**/*.test.ts"
      - "**/*.test.js"
    parallel: true
    timeout: 30000

  integration:
    patterns:
      - "**/*.integration.test.ts"
    timeout: 60000

  e2e:
    patterns:
      - "**/*.e2e.test.ts"
      - "**/*.spec.ts"
    markers:
      - e2e
      - browser

# Environments
environments:
  dev-server:
    type: standalone
    server:
      command: npm run dev
      port: 3000
      readyPattern: "ready on"

  browser-testing:
    type: browser
    server:
      command: npm run dev
      port: 3000
    tunnel:
      port: 3000
      region: us

# Jest configuration
jest:
  configFile: jest.config.js
  coverage: false
  maxWorkers: auto

# Playwright configuration
playwright:
  configFile: playwright.config.ts
  headed: false
  browser: chromium
  retries: 1

CLI Commands

systemeval-ts test

Run tests using the configured adapter.

Options:
  -c, --category <category>  Test category to run
  -p, --parallel             Run tests in parallel
  -x, --failfast             Stop on first failure
  -v, --verbose              Verbose output
  --coverage                 Collect coverage
  --json                     Output results as JSON
  -t, --template <name>      Output template (summary, markdown, json, junit)
  --browser                  Run Playwright tests
  --headed                   Run browser tests in headed mode
  --adapter <name>           Adapter to use (jest, playwright)
  --timeout <ms>             Test timeout in milliseconds
  --config <path>            Path to config file

systemeval-ts init

Initialize a new configuration file.

Options:
  -f, --force  Overwrite existing config

systemeval-ts discover

Discover available tests without running them.

Options:
  -c, --category <category>  Test category
  --adapter <name>           Adapter to use
  --json                     Output as JSON

systemeval-ts list

List available resources.

systemeval-ts list adapters      # List available adapters
systemeval-ts list templates     # List output templates
systemeval-ts list categories    # List configured categories
systemeval-ts list environments  # List configured environments

Adapters

Jest Adapter

Runs tests using Jest with JSON output parsing.

import { createJestAdapter } from 'systemeval-ts';

const adapter = createJestAdapter('/path/to/project', {
  configFile: 'jest.config.js',
  testMatch: ['**/*.test.ts'],
  coverage: true,
  maxWorkers: 4,
});

const result = await adapter.execute({
  parallel: true,
  failfast: false,
  verbose: true,
});

Playwright Adapter

Runs browser tests using Playwright.

import { createPlaywrightAdapter } from 'systemeval-ts';

const adapter = createPlaywrightAdapter('/path/to/project', {
  configFile: 'playwright.config.ts',
  project: 'chromium',
  headed: false,
  timeout: 30000,
  retries: 2,
});

const result = await adapter.execute({
  parallel: true,
});

Environments

Standalone Environment

Manages a single background process (e.g., development server).

import { createStandaloneEnvironment } from 'systemeval-ts';

const env = createStandaloneEnvironment('dev-server', {
  command: 'npm run dev',
  port: 3000,
  readyPattern: /ready on/i,
  readyTimeout: 30000,
});

await env.setup();
await env.waitReady();
// Run tests...
await env.teardown();

Ngrok Environment

Creates an ngrok tunnel for exposing local services.

import { createNgrokEnvironment } from 'systemeval-ts';

const env = createNgrokEnvironment('tunnel', {
  port: 3000,
  region: 'us',
  authToken: process.env.NGROK_AUTHTOKEN,
});

await env.setup();
await env.waitReady();
console.log(`Tunnel URL: ${env.getTunnelUrl()}`);
await env.teardown();

Browser Environment

Composite environment for browser testing with optional server and tunnel.

import { createBrowserEnvironment } from 'systemeval-ts';

const env = createBrowserEnvironment('e2e', {
  testRunner: 'playwright',
  workingDir: '/path/to/project',
  server: {
    command: 'npm run dev',
    port: 3000,
  },
  tunnel: {
    port: 3000,
  },
  playwright: {
    headed: false,
  },
});

await env.setup();
await env.waitReady();
const result = await env.runTests();
await env.teardown();

Templates

Built-in output templates for formatting results:

• summary - Console-friendly summary with pass/fail counts
• markdown - Markdown report for documentation
• json - Raw JSON output
• junit - JUnit XML format for CI integration
• test-result - Compact single-line result

import { renderTestResult, renderEvaluation } from 'systemeval-ts';

// Render test result
const output = renderTestResult(result, 'summary');
console.log(output);

// Register custom template
import { registerTemplate } from 'systemeval-ts';

registerTemplate('custom', `
Tests: {{ passed }} passed, {{ failed }} failed
Duration: {{ duration }}s
`);

Core Types

TestResult

interface TestResult {
  passed: number;
  failed: number;
  errors: number;
  skipped: number;
  duration: number;
  failures: TestFailure[];
  verdict: Verdict;
  exitCode: number;
  parsedFrom: string;
}

Verdict

enum Verdict {
  PASS = 'PASS',
  FAIL = 'FAIL',
  ERROR = 'ERROR',
}

TestItem

interface TestItem {
  id: string;
  name: string;
  path: string;
  line?: number;
  markers: string[];
  suite?: string;
}

Extending

Custom Adapters

import { BaseAdapter, registerAdapter, TestResult, TestItem } from 'systemeval-ts';

class CustomAdapter extends BaseAdapter {
  readonly name = 'custom';

  async validateEnvironment(): Promise<boolean> {
    // Check required tools exist
    return true;
  }

  async discover(): Promise<TestItem[]> {
    // Return discovered tests
    return [];
  }

  async execute(): Promise<TestResult> {
    // Run tests and return results
    return {
      passed: 0,
      failed: 0,
      errors: 0,
      skipped: 0,
      duration: 0,
      failures: [],
      verdict: Verdict.PASS,
      exitCode: 0,
      parsedFrom: 'custom',
    };
  }

  async getAvailableMarkers(): Promise<string[]> {
    return ['custom'];
  }
}

// Register the adapter
registerAdapter('custom', (dir) => new CustomAdapter(dir));

API Reference

Adapters

| Export | Description | |--------|-------------| | BaseAdapter | Abstract base class for adapters | | JestAdapter | Jest test framework adapter | | PlaywrightAdapter | Playwright browser testing adapter | | createAdapter(name, dir, config?) | Create adapter by name | | registerAdapter(name, factory) | Register custom adapter | | listAdapters() | List registered adapters |

Environments

| Export | Description | |--------|-------------| | Environment | Abstract base class | | StandaloneEnvironment | Single process environment | | NgrokEnvironment | Ngrok tunnel environment | | BrowserEnvironment | Composite browser testing environment |

Config

| Export | Description | |--------|-------------| | loadConfig(path) | Load and validate config file | | searchConfigFile(dir) | Find config file in directory tree | | validateConfig(config) | Validate config with Zod |

Templates

| Export | Description | |--------|-------------| | renderTestResult(result, template) | Render test result | | renderEvaluation(result, template) | Render evaluation result | | registerTemplate(name, content) | Register custom template | | listBuiltinTemplates() | List available templates |

License

MIT