@theokit/sdk-cache

Semantic LLM response cache for @theokit/sdk. Vector + FTS hybrid lookup, in-memory + JSON file stores. Integrates with Agent via the Plugin protocol — zero coupling to the agent kernel.

Extracted from @theokit/[email protected] as part of the SDK 2.0 package split (ADRs D249-D266).

Install

pnpm add @theokit/sdk @theokit/sdk-cache

Quick start

import { Agent } from "@theokit/sdk";
import { Cache } from "@theokit/sdk-cache";

const cache = Cache.semantic({
  embedder: myEmbedderRuntime,    // EmbeddingRuntime (D11)
  threshold: 0.85,
  ttl: { default: "1h", exclude: /weather|today|now/i },
  namespace: "my-app",
  modelId: "openai/gpt-4o-mini",
});

const agent = await Agent.create({
  model: { id: "openai/gpt-4o-mini" },
  plugins: [cache.asPlugin()],
});

await agent.send("What is the capital of France?");   // miss → LLM
await agent.send("Tell me the capital of France");    // semantic hit

API

Cache.semantic(options): Cache

Constructs a semantic cache instance with vector similarity + FTS hybrid lookup.

| Option | Type | Default | Notes | |---|---|---|---| | embedder | CacheEmbedderRuntime | (required) | Minimal subset of EmbeddingRuntime the cache uses. | | threshold | number (0-2) | 0.85 | Cosine similarity threshold for hit/miss. | | ttl | string \| number \| CacheTTLConfig | (none) | TTL config — "1h", 30 * 60, or { default, exclude }. | | namespace | string | "default" | Logical bucket for keys. | | modelId | string | (required) | LLM model identifier (used in cache key composition). | | store | "memory" \| { backend: "json", dir } | "memory" | Persistence backend. |

cache.asPlugin(): Plugin

Returns a Plugin compatible with Agent.create({ plugins }). The plugin hooks preUserSend (cache lookup) and postAssistantReply (cache store).

cache.stats()

Returns runtime stats — { hits, misses, entries, namespace }.

cache.clear()

Clears the in-memory state. For the JSON file store, deletes the persisted state on disk.

Errors

• CacheEmbedderError — embedder runtime failed (rate-limited, network error, etc.). Agent run continues without cache.
• CacheInvalidTtlError — TTL string failed parse (e.g., "-5m" or "abc"). Thrown pre-construction.

How it fits with @theokit/sdk

• Foundation: definePlugin + Plugin types come from @theokit/sdk.
• Persistence primitives: atomicWriteText, PersistenceSchema come from @theokit/sdk/internal/persistence (semver-exempt internal API).
• No kernel coupling: the cache subsystem never imports from @theokit/sdk/internal/runtime or the agent loop. Integration is exclusively through the plugin hooks.

Migration from @theokit/[email protected]

Before (1.x):

import { Cache, Agent } from "@theokit/sdk";

After (2.x):

import { Agent } from "@theokit/sdk";
import { Cache } from "@theokit/sdk-cache";

See docs/migration/1-x-to-2-0.md in the monorepo root.

License

Apache-2.0.