@holokai/holo-test

Conformance testing framework for Holo provider and datastore plugins.

The framework owns the test logic. Plugins ship only data: per-protocol scenario files plus an optional minimal SDK adapter. A single runConformance() call discovers every installed @holokai/holo-provider-* and @holokai/holo-datastore-* package, loads its scenarios, and registers translator + wire + audit + pipeline + sdk vitest blocks.

Installation

pnpm add @holokai/holo-test

Peer dependencies: vitest ^3.0.0, tsyringe ^4.10.0

Quick Start

A single test file at the repo root (or in any package that depends on the plugins) is enough:

// tests/conformance.test.ts
import 'reflect-metadata';
import {runConformance} from '@holokai/holo-test';

await runConformance();

runConformance() calls vitest.describe internally; running this file with vitest run produces one suite per discovered plugin, with one describe per scenario.

Testing Modes

| Mode | What it tests | Triggered when | |----------------|--------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------| | Translator | Plugin translator round-trips between native and Holo canonical types | scenario.request and/or scenario.response set | | Wire | Provider events serialize to the expected HTTP wire (status, headers, body) | scenario.wire set + scenario.providerChunks populated | | Audit | Provider events produce the expected ProviderResponse audit shape | scenario.audit set + scenario.providerChunks populated | | Pipeline | End-to-end provider events → wire chunks → text accumulation; chunk-level shape and done terminator are asserted | scenario.pipeline === true + scenario.providerChunks + scenario.expectedText set | | SDK | Real provider SDK client → fixture HTTP server → expected SDK result | scenario.sdk set + tests/sdk-adapter.ts exists | | Datastore | Datastore plugin connects, writes a request/response/cost row, can be queried | tests/datastore-config.ts exists in the datastore plugin | | Live | Real provider API call; asserts protocol.streamEventSequence via assertEventSequence | holo-test-live vitest project; gated on isProviderAvailable(family) |

Modes activate independently — a scenario can exercise just wire+audit without setting sdk or pipeline.

Pipeline shape assertions

When scenario.pipeline === true, the framework collects every chunk delivered to publishChunk during the run.

• If scenario.expectedPipelineChunks is provided, the collected sequence must match exactly.
• Otherwise, the soft path checks that the chunk count is > 0 and the final chunk carries done: true — enough to catch wire-adapter regressions that drop the terminator.

Scenarios

Place scenarios under tests/data/{protocol}/{name}.scenario.ts in a plugin. Each file default-exports a HoloScenario:

import type {HoloScenario} from '@holokai/holo-test';

const scenario: HoloScenario = {
    name: 'simple chat completion',
    protocol: 'openai.chatCompletions',
    streaming: false,

    // Translator round-trip (optional)
    request: {
        native: {model: 'gpt-4o', messages: [{role: 'user', content: 'Hi'}]},
        holo: {model: 'gpt-4o', messages: [{role: 'user', content: 'Hi'}]},
    },
    response: {
        native: {/* native ChatCompletion */},
        holo: {/* partial HoloResponse */},
    },

    // Provider event sequence consumed by wire/audit/pipeline/sdk
    providerChunks: [/* native chunk(s) */],
    expectedText: 'Hello, world!',

    // Wire-conformance
    wire: {
        expectedStatus: 200,
        expectedHeaders: {'content-type': 'application/json'},
        expectedWire: ['{"id":"chatcmpl-..."}'],
    },

    // Audit-conformance
    audit: {
        access_model: 'gpt-4o',
        input_tokens: 10,
        output_tokens: 5,
        status: 'success',
    },

    // Pipeline opt-in
    pipeline: true,

    // SDK round-trip (requires tests/sdk-adapter.ts)
    sdk: {
        request: {model: 'gpt-4o', messages: [{role: 'user', content: 'Hi'}]},
        expectedResult: {choices: [{message: {content: 'Hello, world!'}}]},
    },
};

export default scenario;

Loader Invariants

• Directory name must equal scenario.protocol — the loader throws otherwise. So a scenario with protocol: 'claude.messages' must live under tests/data/claude.messages/.
• Files matching *.scenario.ts (or .scenario.js) are picked up automatically; no central registration needed.
• The default export (or named scenario export) is what the loader reads.
• Every scenario's protocol must appear in the plugin's declared protocols map; mismatches throw at discovery time.

Canonical streaming fixtures

packages/holo-test/holo/streaming/*.holo.ts declares the canonical Holo-pivot streaming sequences used by registerStreamingPerPlugin and registerStreamingCross. Each fixture opens with a message_start chunk and closes with a message_stop chunk, mirroring real provider SSE lifecycle. Per-plugin streaming round-trip is strict: translators must emit and consume those lifecycle chunks to round-trip cleanly.

A plugin that doesn't yet emit/consume native equivalents of message_start / message_stop (e.g. Claude message_start envelopes, OpenAI [DONE] terminators, Bedrock messageStart/messageStop) should pin its scenario with expectedFailure: { toHolo: true, fromHolo: true, reason: '...' } until the gap is closed.

Strip helpers

registerStreamingPerPlugin and registerStreamingCross strip volatile fields before comparing chunk arrays:

| Helper | Strips | Where applied | |-------------------------|--------------------------------------------------------------------|----------------------------------------| | stripProviderMetadata | delta.provider, delta.provider_delta (translator pass-through) | Per-plugin round-trip + cross-provider | | stripCrossVolatile | Top-level id / created / model, message-level id / model | Cross-provider only |

Per-plugin (same-provider in/out) round-trip stays strict on id / created / model — a translator that drops or fabricates them now fails the round-trip immediately rather than being masked.

Streaming event sequence assertion

assertEventSequence(events, sequence) walks the ProviderEvent stream and verifies it satisfies the protocol's declared streamEventSequence ({ ordered: string[], repeatable: string[] }). It returns an AssertionError for missing required events, out-of-order non-repeatable events, or skipped required events; it is exercised via runLiveTest (real provider call), not in the offline conformance run. The native event-type extractor handles both top-level .type shapes (Claude, OpenAI Responses, Gemini structured) and discriminator-key shapes (Bedrock-style {contentBlockDelta: {...}}).

Pinning Known Losses

When a translator/wire/audit asymmetry is known and accepted (e.g., a provider drops a Holo content type that has no native equivalent), record it on the scenario rather than silencing the test:

scenario.expectedFailure = {
    response: {fromHolo: true, roundTrip: true}, // Holo→native loses something
    reason: 'Claude response blocks have no equivalent for Holo image content',
};

Failing-as-expected modes flip to it.fails, so CI still fails if the scenario unexpectedly starts passing.

SDK Adapter

If you want to exercise SDK round-trip (real provider SDK client → fixture HTTP server → assert), drop a tests/sdk-adapter.ts in the plugin:

import OpenAI from 'openai';
import type {HoloScenario, ScenarioSdkAdapter} from '@holokai/holo-test';

const adapter: ScenarioSdkAdapter = {
    family: 'openai',

    async call(scenario: HoloScenario, port: number) {
        const client = new OpenAI({apiKey: 'test', baseURL: `http://localhost:${port}/v1`});
        return client.chat.completions.create(scenario.sdk!.request as never);
    },

    routes(scenario: HoloScenario) {
        if (scenario.protocol === 'openai.chatCompletions') {
            return {method: 'POST', path: '/v1/chat/completions'};
        }
        return undefined;
    },
};

export default adapter;

The runner only loads sdk-adapter.ts when at least one scenario sets sdk, so plugins that don't yet care about SDK round-trip can omit the file. See plugins/holo-provider-openai/tests/sdk-adapter.ts for the canonical example.

Datastore Configs

Datastore plugins (@holokai/holo-datastore-*) supply a tests/datastore-config.ts instead of scenarios:

import type {DatastoreConfig} from '@holokai/holo-test';

const config: DatastoreConfig = {
    connectionConfig: {/* host, port, auth, schema, ... */},
    requestTable: 'provider_requests',
    responseTable: 'provider_responses',
    costTable: 'provider_response_costs',
    cleanup: async (insertedIds) => {/* delete inserted rows */},
};

export default config;

runConformance() exercises connect, write, and read on each datastore plugin that has a config file.

Authoring a New Plugin Test

1. Add tests/setup.ts with a single line: import 'reflect-metadata';
2. Drop one or more tests/data/{protocol}/*.scenario.ts files (default-export a HoloScenario).
3. (Optional) add tests/sdk-adapter.ts exporting a default ScenarioSdkAdapter if you want SDK round-trip coverage.
4. Make sure the plugin's package depends on @holokai/holo-test (devDependency) and that the repo-level runConformance() test file picks up the new package via discoverPlugins().

plugins/holo-provider-claude/tests/data/claude.messages/simple.scenario.ts is a good template for translator + wire + audit coverage; plugins/holo-provider-openai/tests/sdk-adapter.ts is the canonical adapter example.

Composing Manually

For unusual cases — e.g. a single scenario you want to register outside the auto-discovery flow — the per-mode register functions are exported:

import {
    loadPlugin,
    loadScenarios,
    registerTranslatorTests,
    registerWireTests,
    registerAuditTests,
    registerPipelineTests,
    registerSdkTests,
} from '@holokai/holo-test';

const plugin = await loadPlugin('openai');
const scenarios = await loadScenarios('openai');

for (const scenario of scenarios.filter(s => s.name === 'tool calling')) {
    registerTranslatorTests(plugin.translator, scenario);
    registerWireTests(plugin, scenario);
    registerAuditTests(plugin, scenario);
    registerPipelineTests('openai', plugin, scenario);
    registerSdkTests(plugin, scenario, undefined);
}

Prefer runConformance() for normal use; the manual path is an escape hatch.

Live Testing

Tests against real provider APIs (separate from conformance — no fixture server, real network). runLiveTest loads the plugin, makes a real call, collects the resulting ProviderEvent stream, and runs assertEventSequence against the protocol's declared streamEventSequence. Failures surface in the returned errors[].

import {runLiveTest, isProviderAvailable} from '@holokai/holo-test';

if (isProviderAvailable('openai')) {
    const result = await runLiveTest('openai', /* streaming */ true);
    console.log(result.passed, result.errors);
}

This package's own tests/live/live.test.ts exercises runLiveTest against every plugin family in both modes; it runs under the holo-test-live vitest project, which pnpm run test:live chains after app-aio and sdk-live. Tests auto-skip when the family's API key (or OLLAMA_HOST) isn't set.

Environment variables for live tests:

• OPENAI_API_KEY, ANTHROPIC_API_KEY (alias for CLAUDE_API_KEY), GOOGLE_API_KEY (alias for GEMINI_API_KEY), BEDROCK_API_KEY, OLLAMA_HOST
• OPENAI_TEST_MODEL, CLAUDE_TEST_MODEL, etc. (override default test models)

Helpers

import {
    loadPlugin,                  // Load and initialize a plugin by family name
    loadScenarios,               // Load scenarios for a single plugin
    loadSdkAdapter,              // Load tests/sdk-adapter.ts for a plugin
    loadDatastoreConfig,         // Load tests/datastore-config.ts for a datastore plugin
    parseSseBody,                // Parse SSE response body into frames
    parseNdjsonBody,             // Parse NDJSON response body
    createMockFetch,             // Mock fetch for unit tests
    createSseMockFetch,          // Mock SSE streaming responses
    collectStreamText,           // Extract all text from a stream
    collectStreamEvents,         // Collect all events from a stream
    discoverProviders,           // Discover available providers via gateway
    assertTestResult,            // Assert a TestResult passed (throws on failure)
    createTestPlugin,            // Build a temporary plugin tarball for upload/install tests
    packPluginFixture,           // Pack an existing plugin directory into a tarball
} from '@holokai/holo-test';

createTestPlugin / packPluginFixture are for runtime plugin tests (upload/install/lifecycle), not for conformance scenarios.

License

MIT