jest-fuzzy

LLM-powered fuzzy matchers for Jest.

Why?

Testing non-deterministic outputs (like LLM responses, generated content, or natural language) with exact string matching is brittle. jest-fuzzy extends Jest with semantic matchers that use lightweight LLMs to judge whether outputs match expectations.

// Instead of this (brittle):
expect(response).toBe("Hello! How can I help you today?");

// Do this (flexible):
await expect(response).toSemanticallyMatch("Hi, how may I assist you?");

Installation

npm install jest-fuzzy --save-dev

That's it! No additional dependencies required. jest-fuzzy uses Anthropic, OpenAI, and Google REST APIs directly.

Quick Start

1. Configure your API key (choose one method):

Option A: Pass it directly (recommended for simplicity):

// jest.setup.ts
import { configure } from "jest-fuzzy";

configure({
  apiKeys: {
    anthropic: "your-api-key-here",
  },
});

Option B: Use environment variables:

export ANTHROPIC_API_KEY=your-key-here
# or GOOGLE_API_KEY for Gemini
# or OPENAI_API_KEY for OpenAI

// jest.setup.ts
import "jest-fuzzy";

2. Reference the setup file in your Jest config:

// jest.config.js
module.exports = {
  setupFilesAfterEnv: ["./jest.setup.ts"],
};

3. Use the matchers in your tests:

describe("chatbot", () => {
  it("responds with a greeting", async () => {
    const response = await chatbot.greet();
    await expect(response).toSemanticallyMatch("Hello, how can I help?");
  });

  it("provides helpful responses", async () => {
    const response = await chatbot.answer("What is 2+2?");
    await expect(response).toSatisfy("contains the number 4 and is polite");
  });
});

API Reference

configure(options?)

Configures jest-fuzzy with your preferred model and API keys.

import { configure } from "jest-fuzzy";

configure({
  model: "claude-haiku-4-5", // or "gemini-3-flash-preview" or "gpt-5-nano"
  apiKeys: {
    anthropic: "your-anthropic-key", // for Claude models
    google: "your-google-key", // for Gemini models
    openai: "your-openai-key", // for OpenAI models
  },
});

| Option | Type | Default | Description | | ------------------- | -------- | --------------------------- | ------------------------------------------------------------ | | model | string | "claude-haiku-4-5" | The LLM model to use for judging | | apiKeys | object | - | API keys for providers (falls back to environment variables) | | apiKeys.anthropic | string | ANTHROPIC_API_KEY env var | API key for Claude models | | apiKeys.google | string | GOOGLE_API_KEY env var | API key for Gemini models | | apiKeys.openai | string | OPENAI_API_KEY env var | API key for OpenAI models |

toSemanticallyMatch(expected, options?)

Tests whether the received string has the same semantic meaning as the expected string.

await expect("Hello! How can I help you today?").toSemanticallyMatch(
  "Hi, how may I assist you?"
);

Options:

| Option | Type | Default | Description | | ----------- | ----------------------- | --------- | ------------------------------------------ | | threshold | "strict" | "loose" | "loose" | How strictly to judge semantic equivalence | | context | string | - | Additional context to help the LLM judge |

Examples:

// Basic usage
await expect("The sky is blue").toSemanticallyMatch(
  "The color of the sky is blue"
);

// With strict threshold
await expect(response).toSemanticallyMatch("The capital of France is Paris", {
  threshold: "strict",
});

// With context
await expect("Paris").toSemanticallyMatch("The capital of France", {
  context: "Testing a geography Q&A bot asking about France's capital",
});

// Negation
await expect("I love sunny weather").not.toSemanticallyMatch(
  "I hate sunny weather"
);

toSatisfy(criteria)

Tests whether the received string satisfies the given natural language criteria.

await expect(summary).toSatisfy(
  "mentions climate change and includes at least one statistic"
);

Examples:

// Testing content
await expect(article).toSatisfy("discusses renewable energy");

// Testing tone
await expect(response).toSatisfy("is polite and professional in tone");

// Testing structure
await expect(output).toSatisfy("is valid JSON containing a 'name' field");

// Testing safety
await expect(response).not.toSatisfy(
  "contains profanity or offensive language"
);

Configuration

API Keys

You can provide API keys in two ways:

1. Direct configuration (recommended):

configure({
  apiKeys: {
    anthropic: process.env.MY_ANTHROPIC_KEY, // or hardcoded string
  },
});

2. Environment variables (fallback):

| Model | Environment Variable | | ------------------------ | -------------------- | | claude-haiku-4-5 | ANTHROPIC_API_KEY | | gemini-3-flash-preview | GOOGLE_API_KEY | | gpt-5-nano | OPENAI_API_KEY |

Direct configuration takes precedence over environment variables.

Supported Models

| Model ID | Provider | Description | | ------------------------ | --------- | ------------------------------------------- | | claude-haiku-4-5 | Anthropic | Fast, cost-effective Claude model (default) | | gemini-3-flash-preview | Google | Fast Gemini model | | gpt-5-nano | OpenAI | Lightweight GPT model |

TypeScript Support

jest-fuzzy includes full TypeScript support. The matchers are automatically typed when you import the library:

import "jest-fuzzy";

// Types are automatically available
await expect("hello").toSemanticallyMatch("hi");
await expect("hello").toSatisfy("is a greeting");

Error Messages

When tests fail, jest-fuzzy includes the LLM's reasoning to help you understand why:

Expected "The weather is nice" to semantically match "It's raining heavily"

LLM reasoning: These statements have opposite meanings. The first indicates
pleasant weather conditions, while the second describes heavy rainfall.

Best Practices

1. Use semantic matching for equivalent meanings, not for checking specific content:

   // Good: checking semantic equivalence
   await expect(greeting).toSemanticallyMatch("Hello, how can I help?");
   
   // Better for specific content: use toSatisfy
   await expect(response).toSatisfy("mentions the user's name 'John'");

2. Provide context when the comparison needs domain knowledge:

   await expect("42").toSemanticallyMatch("The answer", {
     context: "Testing a Hitchhiker's Guide to the Galaxy trivia bot",
   });

3. Use threshold: "strict" when precision matters:

   await expect(legalText).toSemanticallyMatch(expectedDisclaimer, {
     threshold: "strict",
   });

4. Consider test runtime: Each matcher makes an API call, so use them judiciously in large test suites.

Author

Christian Cantrell

License

MIT