Kurt Adapter for Caching

Kurt is a TypeScript library by Formula.Monks that wraps AI SDKs, making it easy to build structured LLM-based applications (RAG, agents, etc) that work with any LLM that supports structured output (via function calling features).

This package implements an adapter for Kurt that caches responses to disk. This is most useful for testing and development, for:

• ensuring determinism of the test data and code paths
• avoiding unnecessary AI usage costs for repetitive requests
• allowing for running existing tests without requiring an API key for the AI service

The cache entries are YAML files, which can be easily inspected and modified for your test cases, as well as checked into your code repository for meaningful code review.

Read here for more information about Kurt.

Examples

This example code shows how to set up and use KurtCache with KurtOpenAI (though it also works with other adapters).

import { Kurt } from "@formula-monks/kurt"
import { KurtCache } from "@formula-monks/kurt-cache"
import { KurtOpenAI } from "@formula-monks/kurt-open-ai"
import OpenAI from "openai"
import { z } from "zod"

const cacheAdapter = new KurtCache(
  // This is the directory where the cache will be stored.
  `${__dirname}/.kurt-cache`,
  // This is the cache prefix. It should identify this adapter configuration,
  // If the prefix changes, prior matching cache entries will no longer match.
  "openai-gpt-3.5-turbo-0125",
  // This function will only be run the first time we encounter a cache miss.
  () =>
    new KurtOpenAI({
      openAI: new OpenAI(),
      model: "gpt-3.5-turbo-0125",
    })
)

const kurt = new Kurt(cacheAdapter)

const schema = z.object({ say: z.string().describe("A single word to say") })
const stream1 = kurt.generateStructuredData({ prompt: "Say hello!", schema })
const stream2 = kurt.generateStructuredData({ prompt: "Say hello!", schema })
const stream3 = kurt.generateStructuredData({ prompt: "Say hi!", schema })

console.log((await stream1.result).data) // (cache miss on first run)
console.log((await stream2.result).data) // (always cached; identical to prior)
console.log((await stream3.result).data) // (cache miss on first run)