@skill-kit/test

Testing framework for Skill Kit - test skill triggers, mock LLM responses, and generate coverage reports.

Installation

npm install @skill-kit/test
# or
pnpm add @skill-kit/test

Quick Start

1. Create a test file

Create a file my-skill.skilltest.ts:

import { defineTests } from '@skill-kit/test';

export default defineTests({
  skill: './SKILL.md',
  
  cases: [
    {
      name: 'should trigger on exact match',
      input: 'test something for me',
      shouldTrigger: true,
    },
    {
      name: 'should not trigger on unrelated input',
      input: 'what is the weather today',
      shouldTrigger: false,
    },
    {
      name: 'should trigger with fuzzy match',
      input: 'tset somethng',
      shouldTrigger: true,
      matchType: 'fuzzy',
      minConfidence: 0.7,
    },
  ],
});

2. Run tests

npx skill-test run

Or with watch mode:

npx skill-test watch

Configuration

Create .skilltestrc.json in your project root:

{
  "testDir": "tests",
  "include": ["**/*.skilltest.ts"],
  "exclude": ["**/node_modules/**"],
  "timeout": 10000,
  "coverage": {
    "enabled": true,
    "threshold": 80,
    "reporters": ["text", "json", "html"]
  },
  "reporters": ["console", "json"]
}

API Reference

Test Definition

import { defineTests } from '@skill-kit/test';

export default defineTests({
  skill: './SKILL.md',
  description: 'My skill tests',
  timeout: 5000,
  
  beforeAll: async () => { /* setup */ },
  afterAll: async () => { /* cleanup */ },
  beforeEach: async () => { /* per-test setup */ },
  afterEach: async () => { /* per-test cleanup */ },
  
  cases: [
    {
      name: 'test name',
      input: 'user input',
      shouldTrigger: true,
      expectedSkill: 'skill-name',
      matchType: 'exact', // 'exact' | 'contains' | 'fuzzy' | 'regex'
      minConfidence: 0.8,
      tags: ['smoke', 'regression'],
      skip: false,
      only: false,
      timeout: 1000,
      setup: async () => { /* test-specific setup */ },
      teardown: async () => { /* test-specific cleanup */ },
    },
  ],
});

Matcher Functions

import { matchExact, matchContains, matchFuzzy, matchRegex, createMatcher } from '@skill-kit/test';

// Exact match
const result1 = matchExact('test something', ['test something', 'run tests']);

// Contains match
const result2 = matchContains('please test something', ['test something']);

// Fuzzy match (with typo tolerance)
const result3 = matchFuzzy('tset somthing', ['test something'], 0.7);

// Regex match
const result4 = matchRegex('test hello', ['/test\\s+\\w+/i']);

// Combined matcher
const matcher = createMatcher({
  config: {
    exact: true,
    contains: true,
    fuzzy: true,
    regex: true,
    fuzzyThreshold: 0.7,
  },
});
const result = matcher.match('input', ['trigger1', 'trigger2']);

Mock LLM Provider

import { createMockProvider } from '@skill-kit/test';

const mock = createMockProvider({
  responses: {
    'test-skill': 'Mocked response',
  },
  templates: {
    'dynamic-skill': (input) => `Processed: ${input}`,
  },
  errors: {
    'error-skill': new Error('Simulated error'),
  },
  delay: 100,
});

// Use in tests
const response = await mock.respond('test-skill', 'input');

// Dynamic configuration
mock.setResponse('new-skill', 'New response');
mock.setDelay(200);
mock.reset();

Coverage Collection

import { createCoverageCollector, generateCoverageReport } from '@skill-kit/test';

const collector = createCoverageCollector();

// Register skills
collector.registerSkill(skillFile);

// Record tested triggers
collector.recordTriggerTest('/path/SKILL.md', 'trigger1');

// Generate report
const report = collector.getReport(80); // 80% threshold

// Write reports
await generateCoverageReport(report, ['text', 'json', 'html'], './coverage');

Test Runner

import { createTestRunner, runTests } from '@skill-kit/test';

// Create runner with custom config
const runner = createTestRunner({
  config: {
    timeout: 5000,
    failFast: true,
    verbose: true,
  },
  onTestStart: (test) => console.log(`Running: ${test.name}`),
  onTestEnd: (result) => console.log(`${result.passed ? 'PASS' : 'FAIL'}: ${result.name}`),
});

// Run single suite
const result = await runner.run(suite);

// Run from file
const result2 = await runner.runFile('./my-skill.skilltest.ts');

// Run all tests
const result3 = await runner.runAll('**/*.skilltest.ts');

CLI Commands

Run Tests

skill-test run [options]

Options:
  -c, --config <path>    Path to config file
  -d, --test-dir <dir>   Test directory
  -t, --timeout <ms>     Test timeout
  --coverage             Enable coverage
  -r, --reporter <type>  Reporter (console, json, html)
  -v, --verbose          Verbose output
  --fail-fast            Stop on first failure
  -p, --pattern <glob>   Test file pattern

Watch Mode

skill-test watch [options]

Options:
  -c, --config <path>    Path to config file
  -d, --test-dir <dir>   Test directory
  -v, --verbose          Verbose output

Match Types

| Type | Description | Confidence | |------|-------------|------------| | exact | Exact match (case-insensitive) | 1.0 | | contains | Input contains trigger | 0.6-0.9 | | fuzzy | Levenshtein distance match | 0.7-0.99 | | regex | Regular expression match | 0.95 |

License

MIT