@holokai/holo-test

Conformance testing framework for Holo provider plugins. Provides fixture-driven testers, vitest contracts, a fluent DSL, and a CLI.

Installation

npm install @holokai/holo-test

Peer dependencies: vitest ^3.0.0, tsyringe ^4.10.0

Quick Start

import {describe, it} from 'vitest';
import {loadFixtures, runWireContract} from '@holokai/holo-test';

const fixtures = loadFixtures('./tests/fixtures', {plugin: 'openai'});

describe('openai wire conformance', () => {
    for (const fixture of fixtures) {
        it(fixture.name, async () => {
            await runWireContract('openai', fixture);
        });
    }
});

Testing Modes

| Mode | What it Tests | Function | |---------------|------------------------------------------------------------------------------------|----------------------------------------------| | Wire | Provider events serialize to correct HTTP wire chunks (SSE/JSON) | testWire() / runWireContract() | | Audit | Worker requests produce correct ProviderRequest/ProviderResponse audit records | testAudit() / runAuditContract() | | Pipeline | End-to-end: payload in, events out, text accumulated, done event emitted | testPipeline() / runPipelineContract() | | RoundTrip | SDK request → HTTP fixture server → SDK response matches expected output | testRoundTrip() / runRoundTripContract() |

Fixtures

Create .fixture.ts files that export a FixtureScenario:

import type {FixtureScenario} from '@holokai/holo-test';

const fixture: FixtureScenario = {
    name: 'simple chat completion',
    plugin: 'openai',
    protocol: 'openai.chatCompletions',
    streaming: false,

    providerChunks: [/* mock provider response objects */],

    expectedText: 'Hello, world!',
    expectedWire: ['{"id":"chatcmpl-...}'],
    expectedStatus: 200,
    expectedHeaders: {'content-type': 'application/json'},

    expectedAudit: {
        access_model: 'gpt-4o',
        input_tokens: 10,
        output_tokens: 5,
        status: 'success',
    },

    tags: ['chat', 'non-streaming'],
};

export default fixture;

Load fixtures:

import {loadFixtures} from '@holokai/holo-test';

// All fixtures
const all = loadFixtures('./tests/fixtures');

// Filtered
const streaming = loadFixtures('./tests/fixtures', {plugin: 'openai', tag: 'streaming'});

Suite DSL

Fluent builder for running multiple test modes:

import {suite} from '@holokai/holo-test';

await suite()
    .forPlugin('openai')
    .fixtureDir('./tests/fixtures')
    .wire()
    .audit()
    .pipeline()
    .verbose()
    .run();

Vitest Contracts

Import contract functions directly into vitest test files:

import {runWireContract, runAuditContract, runPipelineContract} from '@holokai/holo-test';

describe('openai conformance', () => {
    it('wire: simple chat', async () => {
        await runWireContract('openai', fixture);
    });

    it('audit: simple chat', async () => {
        await runAuditContract('openai', fixture);
    });

    it('pipeline: simple chat', async () => {
        await runPipelineContract('openai', fixture);
    });
});

CLI

npx holo-test --fixture-dir ./tests/fixtures --plugin openai
npx holo-test --fixture-dir ./tests/fixtures --plugin claude --tag streaming --verbose

Live Testing

Test against real provider APIs:

import {runLiveTest, isProviderAvailable} from '@holokai/holo-test';

if (isProviderAvailable('openai')) {
    const results = await runLiveTest({
        families: ['openai'],
        timeout: 30000,
    });
    console.log(results);
}

Environment variables for live tests:

• OPENAI_API_KEY, CLAUDE_API_KEY, GEMINI_API_KEY, OLLAMA_HOST
• OPENAI_MODEL, CLAUDE_MODEL, etc. (override default test models)

Helpers

import {
    loadPlugin,                  // Load and initialize a plugin by family name
    parseSseBody,                // Parse SSE response body into frames
    parseNdjsonBody,             // Parse NDJSON response body
    createMockFetch,             // Mock fetch for unit tests
    createSseMockFetch,          // Mock SSE streaming responses
    collectStreamText,           // Extract all text from a stream
    collectStreamEvents,         // Collect all events from a stream
    HttpFixtureServer,           // Mock HTTP server serving fixture responses
    createTestWorkerRequest,     // Build a HoloWorkerRequest from a fixture
    createTestEnvelope,          // Build a WorkerResponseEnvelope from a fixture
    discoverProviders,           // Discover available providers via gateway
    assertTestResult,            // Assert a TestResult passed (throws on failure)
} from '@holokai/holo-test';

License

MIT