@llmist/testing

Testing utilities for llmist - mock LLM responses and test agents deterministically.

Installation

npm install -D @llmist/testing

Requires llmist as a peer dependency.

Quick Start

Testing Gadgets

Test gadgets in isolation without any LLM calls:

import { testGadget } from '@llmist/testing';
import { Calculator } from './gadgets';

const result = await testGadget(new Calculator(), {
  operation: 'add',
  a: 5,
  b: 3,
});

expect(result.result).toBe('8');
expect(result.error).toBeUndefined();

Mocking LLM Responses

Use the fluent MockBuilder API to script LLM responses:

import { mockLLM, createMockClient, resetMocks } from '@llmist/testing';
import { LLMist } from 'llmist';

// Set up mock responses
mockLLM()
  .whenMessageContains('hello')
  .returns('Hi there! How can I help?')
  .register();

mockLLM()
  .whenMessageContains('calculate')
  .returnsGadgetCall('Calculator', { operation: 'add', a: 1, b: 2 })
  .register();

// Create agent with mock client
const agent = LLMist.createAgent()
  .withClient(createMockClient())
  .withGadgets(Calculator);

const response = await agent.askAndCollect('hello');
// Returns "Hi there! How can I help?" - no API calls made

// Clean up after tests
resetMocks();

Conditional Mocking

Match responses based on model, provider, or custom conditions:

mockLLM()
  .forModel('gpt-4o')
  .forProvider('openai')
  .whenMessageContains('complex task')
  .returns('Handled by GPT-4o')
  .register();

mockLLM()
  .forModel('haiku')
  .whenMessageContains('complex task')
  .returns('Handled by Haiku')
  .register();

One-Time Responses

Use .once() for responses that should only match once:

mockLLM()
  .whenMessageContains('first')
  .returns('First response')
  .once()
  .register();

mockLLM()
  .whenMessageContains('first')
  .returns('Second response')
  .register();

// First call returns "First response"
// Subsequent calls return "Second response"

API Reference

testGadget(gadget, params, options?)

Test a gadget with given parameters.

mockLLM()

Create a new mock builder with fluent API.

createMockClient()

Create a mock LLMist client that uses registered mocks.

resetMocks()

Clear all registered mocks (call in afterEach).

Documentation

Full documentation at llmist.dev/testing

• Mocking Overview
• Testing Gadgets
• Testing Agents

Related Packages

• llmist - Core library
• @llmist/cli - Command-line interface

License

MIT