prompt-cache-optimizer

Drop-in wrappers for the Anthropic, OpenAI, and Gemini SDKs that make prompt caching effortless. Measure real cache hit rate from the response usage object, attach dollar savings to every call, canonicalize shuffled tools and RAG document order so a "slightly different" payload still hits the cache, and (for Anthropic) auto-place cache_control breakpoints based on observed stability.

Real output from bun run example. Six calls — autoCache marks the system prompt cacheable after observing it twice, calls 3–5 hit the cache (~1569 cached tokens each), and a final deliberate drift triggers the diagnostic showing the exact characters that changed. client.stability() reports system score=0.80 cumulative across the run.

Status: v0.6 — adds OpenAI Responses API wrapper and Gemini auto-cache TTL auto-refresh. Backwards compatible with v0.1–0.5.

Why this exists

All three frontier providers now offer prompt caching:

• Anthropic — 90% discount on the cached portion. Marker-based (cache_control: { type: "ephemeral" }), positional, 4-breakpoint budget, 5-minute TTL.
• OpenAI — 50% discount on the cached portion (75% for o-series). Automatic for prompts ≥ 1024 tokens, no markers — but silently doesn't trigger below the threshold or when your tools array shuffles.
• Gemini — 75% discount on the cached portion. Two modes: implicit (automatic for 2.5+) and explicit (CachedContent API with manual lifecycle).

All three are fragile in similar ways: a misplaced byte, a reshuffled tools array, a TTL expiry, an upstream service that reorders retrieved documents — and your prompt cache silently degrades to a full-price call. The only way to know it's working is to dig into the response usage object yourself.

prompt-cache-optimizer handles all of that for you, with the same surface for every provider.

Install

# Pick the provider(s) you actually use. OpenAI and Gemini SDKs are
# optional peer deps — install only what you need.
npm install prompt-cache-optimizer @anthropic-ai/sdk
npm install prompt-cache-optimizer openai
npm install prompt-cache-optimizer @google/genai

Quick start: Anthropic (auto-placement + auto-reorder)

import { CachedAnthropic } from "prompt-cache-optimizer";

const client = new CachedAnthropic({
  apiKey: process.env.ANTHROPIC_API_KEY!,
  autoCache: true,           // ← let the wrapper place cache_control for you
  autoReorder: true,         // ← canonicalize shuffled tools / RAG docs (v0.3)
  diagnoseMisses: true,      // ← explain what changed when the cache misses
  warnIfHitRateBelow: 0.6,
});

// Use the SDK exactly like normal. No placeBreakpoints() needed.
const response = await client.messages.create({
  model: "claude-sonnet-4-6",
  max_tokens: 1024,
  system: longSystemPrompt,
  messages: conversation,
});

console.log(response.cacheInfo);
// { hit: true, cachedTokens: 8420, uncachedTokens: 312, dollarsSaved: 0.024, ... }

console.log(client.stats());
// { totalCalls: 1, hitRate: 1, totalCachedTokens: 8420, dollarsSaved: 0.024, ... }

console.log(client.stability());
// { entries: [{ segment: 'system', stabilityScore: 1, approxTokens: 2103, ... }], ... }

The first call always misses (that's when the cache is written). Once the wrapper has seen the system prompt twice unchanged, it auto-marks it cacheable and subsequent calls hit. No code changes needed when your prompt shape evolves — auto-placement re-evaluates each call.

Quick start: OpenAI

import { CachedOpenAI } from "prompt-cache-optimizer";

const client = new CachedOpenAI({
  apiKey: process.env.OPENAI_API_KEY!,
  autoReorder: true,         // ← alphabetize tools so shuffled lists still hit cache
  diagnoseMisses: true,
  warnIfHitRateBelow: 0.5,
  // warnIfPromptTooSmall is on by default — surfaces calls below OpenAI's
  // 1024-token automatic-cache minimum so you know why no caching happens.
});

const response = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [
    { role: "system", content: longSystemPrompt },
    { role: "user", content: question },
  ],
  tools: [...],
});

console.log(response.cacheInfo);
// { hit: true, cachedTokens: 7420, uncachedTokens: 312, dollarsSaved: 0.0093, ... }

console.log(client.stats());
// { totalCalls: 5, hitRate: 0.8, totalCachedTokens: 29680, dollarsSaved: 0.037, ... }

OpenAI's cache is automatic for prompts ≥ 1024 tokens — there is no autoCache to enable (and no cache_control markers to place). The wrapper measures from usage.prompt_tokens_details.cached_tokens, attaches per-call savings, accumulates rolling stats, and (with autoReorder: true) canonicalizes the tools array so a shuffled list still hits.

Quick start: Gemini

import { CachedGemini } from "prompt-cache-optimizer";

const client = new CachedGemini({
  apiKey: process.env.GOOGLE_API_KEY!,
  autoReorder: true,
  diagnoseMisses: true,
});

// Implicit caching (Gemini 2.5+ automatic): just call it.
const response = await client.models.generateContent({
  model: "gemini-2.5-flash",
  contents: [{ role: "user", parts: [{ text: question }] }],
  config: { systemInstruction: longSystemInstruction },
});
console.log(response.cacheInfo);
// { hit: true, cachedTokens: 4800, uncachedTokens: 500, dollarsSaved: 0.00045, ... }

// Explicit caching: create a CachedContent and reference it by name.
const cache = await client.caches.create({
  model: "gemini-2.5-flash",
  config: {
    contents: [{ role: "user", parts: [{ text: longContext }] }],
    ttl: "300s",
  },
});

const cached = await client.models.generateContent({
  model: "gemini-2.5-flash",
  contents: [{ role: "user", parts: [{ text: question }] }],
  config: { cachedContent: cache.name },
});
console.log(cached.cacheInfo);
// → hit: true, cachedTokens: (all of longContext)

await client.caches.delete({ name: cache.name }); // clean up when done

Gemini exposes the SDK's caches.create/get/delete/list/update pass-through so you can manage CachedContent lifecycles through the same client. Auto-managed explicit caching (the wrapper creates and refreshes CachedContent objects on its own when prefixes are stable) is on the v0.5 roadmap.

What you get with every provider

Regardless of provider, every wrapped client exposes the same surface:

• response.cacheInfo — { hit, cachedTokens, uncachedTokens, cacheWriteTokens, dollarsSaved, dollarsSpent } on every call.
• client.stats() — rolling aggregate: totalCalls, hitRate, totalCachedTokens, dollarsSaved, etc.
• client.stability() — per-segment stability report so you can debug which part of your prompt is drifting before it costs money.
• client.resetStats() — clears stats and stability history.
• autoReorder — canonicalizes order-insensitive parts of the request so shuffled inputs still hit the cache.
• diagnoseMisses — when the cache misses, attach a human-readable diff explaining what changed in the prefix.
• Built-in pricing tables — override per-instance with pricingOverride.
• Passive warning events via onWarning — never throws, never blocks a request.

Streaming (v0.5)

Every provider client gained a streaming entry point in v0.5. The shape is consistent: an async-iterable you can consume chunk-by-chunk AND a .final() promise that resolves with cacheInfo after the stream ends.

// Anthropic
const stream = client.messages.stream({
  model: "claude-sonnet-4-6",
  max_tokens: 1024,
  system: [{ type: "text", text: longSystemPrompt, cache_control: { type: "ephemeral" } }],
  messages: conversation,
});
for await (const event of stream) {
  if (event.type === "content_block_delta") process.stdout.write(event.delta.text ?? "");
}
const { cacheInfo, raw } = await stream.final();
console.log(`saved $${cacheInfo.dollarsSaved.toFixed(4)} on this stream`);

// OpenAI — pass `stream: true` to the existing create method
const stream = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [...],
  stream: true,
  // The wrapper auto-enables stream_options.include_usage:true so the
  // final chunk's usage object is available for cacheInfo. Without it,
  // OpenAI returns no usage at all in streaming mode.
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices?.[0]?.delta?.content ?? "");
}
const { cacheInfo } = await stream.final();

// Gemini — dedicated generateContentStream method
const stream = await client.models.generateContentStream({
  model: "gemini-2.5-flash",
  contents: [{ role: "user", parts: [{ text: question }] }],
  config: { systemInstruction: longSystemInstruction },
});
for await (const chunk of stream) {
  process.stdout.write(chunk.text ?? "");
}
const { cacheInfo } = await stream.final();

If you don't want incremental chunks, you can skip the iteration entirely — final() will drain the stream itself. Streaming responses participate in the same stats() / stability() / diagnoseMisses pipeline as non-streaming responses.

Auto-managed Gemini explicit caching (v0.5)

Gemini's CachedContent API gives you the deepest cache savings on long context (~75% off cached tokens) but it's the most cumbersome to use — you have to POST a cache resource, get back a name, reference it, then clean it up. v0.5 automates the whole lifecycle:

const client = new CachedGemini({
  apiKey,
  autoCache: true,                  // ← enable auto-managed CachedContent
  autoCacheMinObservations: 2,      // create after 2 stable observations (default)
  autoCacheTtl: 600,                // seconds (default 300)
});

// Just call it normally — the wrapper handles the cache lifecycle.
for (const question of questions) {
  const res = await client.models.generateContent({
    model: "gemini-2.5-flash",
    contents: [{ role: "user", parts: [{ text: question }] }],
    config: { systemInstruction: longSystemInstruction },
  });
  console.log(res.cacheInfo);  // hit:true after the cache is created
}

console.log(client.managedCaches());
// [{ fingerprint: '...', name: 'cachedContents/auto-1', expiresInSeconds: 587, approxTokens: 8420 }]

await client.gc();  // optional — sweep expired entries

How it works:

1. The wrapper observes config.systemInstruction across calls and tracks consecutive-stable counts per fingerprint.
2. After autoCacheMinObservations calls (default 2) with the same instruction, it calls caches.create({ contents: [systemInstruction] }) and stashes the resulting cache name keyed by fingerprint.
3. On subsequent matching calls, it swaps systemInstruction out and cachedContent: name in. Gemini bills the cached portion at ~25% of standard input.
4. When the instruction changes, the previously created cache is best-effort deleted.
5. Failure modes are silent: if Gemini rejects the create (most commonly because the instruction is below the ~32k-token minimum for explicit caching), the wrapper falls back to passing the instruction verbatim — the implicit cache still works.

Explicit user intent always wins. If you pass config.cachedContent yourself, the wrapper leaves it alone.

TTL auto-refresh (v0.6)

In a long-running service, the TTL on a managed CachedContent will eventually expire — at which point the next observation re-creates it. For chatty services this means a quiet drumbeat of caches.create calls just to keep the cache warm.

v0.6 adds proactive TTL refresh. Every time the wrapper reuses a managed cache, it checks how much of the original TTL is left. If the remainder is below autoCacheTtlRefreshFraction (default 0.2 = 20%), it fires a caches.update({ name, config: { ttl } }) to extend it. The update is fire-and-forget; failures are silently swallowed (the cache will just expire naturally and be re-created on the next observation).

const client = new CachedGemini({
  apiKey,
  autoCache: true,
  autoCacheTtl: 600,                  // 10 minutes
  autoCacheTtlRefreshFraction: 0.2,   // refresh when under 2 minutes left (default)
});

// After running for a while:
console.log(client.managedCaches());
// → [{ ..., refreshes: 3 }] — this cache has been extended 3 times.

Set autoCacheTtlRefreshFraction: 0 to disable refresh entirely (caches will just expire and be re-created on next observation — the v0.5 behavior).

OpenAI Responses API (v0.6)

OpenAI's newer Responses API uses a different shape from chat.completions — input instead of messages, input_tokens / input_tokens_details.cached_tokens in usage instead of prompt_tokens / prompt_tokens_details.cached_tokens. v0.6 wraps it with the same drop-in feel as the chat.completions wrapper:

import { CachedOpenAI } from "prompt-cache-optimizer";

const client = new CachedOpenAI({
  apiKey: process.env.OPENAI_API_KEY!,
  autoReorder: true,
});

// Non-streaming
const response = await client.responses.create({
  model: "gpt-4o",
  input: "Summarize the following long doc...",
  tools: [...],   // function-type tools get alphabetized with autoReorder
});
console.log(response.cacheInfo);
// { hit: true, cachedTokens: 8420, dollarsSaved: 0.024, ... }
console.log(response.output_text);

// Streaming
const stream = await client.responses.create({
  model: "gpt-4o",
  input: "Long prompt here...",
  stream: true,
});
for await (const event of stream) {
  if (event.type === "response.output_text.delta") {
    process.stdout.write((event as { delta?: { text?: string } }).delta?.text ?? "");
  }
}
const { cacheInfo } = await stream.final();

Same client.stats(), client.stability(), client.resetStats() work across both APIs — they share one set of internals.

How Anthropic auto-placement decides what to cache

(Anthropic only — OpenAI's cache is automatic and has no markers, and Gemini's explicit CachedContent is managed by you.)

On every call the wrapper:

1. Fingerprints each candidate segment — system, tools, and every cumulative messages[0..N] prefix — using SHA-256 over a canonical form (cache_control markers stripped so they don't affect the hash).
2. Tracks the fingerprint history per segment.
3. Once a segment has been seen unchanged for at least autoCacheMinObservations consecutive calls (default 2), it qualifies for auto-placement.
4. Picks the highest-value placements within Anthropic's 4-breakpoint budget: system first, then tools, then the longest stable message prefix.

You can inspect this state live with client.stability().

Manual breakpoint placement (still supported)

If you want explicit control, placeBreakpoints from v0.1 still works exactly as before. Auto-placement is a no-op whenever you've already marked anything cacheable yourself — your intent is always respected.

import { placeBreakpoints } from "prompt-cache-optimizer";

const { system, messages } = placeBreakpoints({
  system: longSystemPrompt,
  messages: conversation,
  strategy: "after-system",
});

await client.messages.create({ model, max_tokens, system, messages });

Three strategies are available:

• after-system — cache the system prompt (best for RAG and long instructions)
• after-last-assistant — cache the conversation history (best for chat)
• system-and-history — cache both (uses 2 of your 4 breakpoints)

Stats

client.stats();
// {
//   totalCalls: 142,
//   cacheHits: 124,
//   hitRate: 0.873,
//   totalCachedTokens: 1_240_000,
//   totalUncachedTokens: 52_400,
//   totalCacheWriteTokens: 21_000,
//   dollarsSaved: 3.72,
//   dollarsSpent: 1.41,
// }

Cache-miss diagnostics

Enable diagnoseMisses: true and every cache-write-without-read warning gets a structured diff explaining what changed. Example:

new CachedAnthropic({
  apiKey,
  diagnoseMisses: true,
  onWarning: (event) => {
    if (event.code === "cache-write-without-read") {
      console.error(event.message);
      // → "...Detected: system prompt changed at character 1240: ...the docs as of [Tuesday|Wednesday]..."
      console.error(event.detail?.diff);
      // → [{ segment: 'system', summary: '...', detail: { changeIndex: 1240, ... } }]
    }
  },
});

Common things it catches:

• system prompt drift (inserted timestamps, dynamic context)
• tool order changes (v0.3's autoReorder fixes this one automatically)
• retrieved-document reordering (v0.3's autoReorder fixes this one automatically too)
• TTL expiration (cache was fine, then nobody called within 5 minutes)

Auto-reorder (v0.3)

The fastest way to silently break Anthropic's prompt cache is to send the same logical content in a different order. Examples:

• Your tool definitions are pulled from an object — JS preserves insertion order, but two different code paths assemble them differently.
• Your RAG retriever returns the same five documents but ranked differently call to call.
• A user message contains multiple document content blocks shuffled by an upstream service.

Each of those silently degrades to a full-price cache write. Set autoReorder: true and the wrapper canonicalizes the order-insensitive parts of the request before sending:

const client = new CachedAnthropic({
  apiKey,
  autoReorder: true,
  // commonly paired with autoCache so the canonicalized payload also
  // gets cache_control placed on it automatically:
  autoCache: true,
});

What gets reordered:

• Tools — alphabetized by name. Tool order is semantically irrelevant to the model, so this is always safe.
• Content blocks within a message — consecutive runs of same-type "reorderable" blocks (document, image) are sorted by content fingerprint. Text, tool_use, tool_result, and thinking blocks are never moved — they're order-sensitive.
• Leading user-context messages — a leading run of user messages whose content is purely document/image blocks (the classic RAG pattern) is sorted by content fingerprint. The scan stops at the first message that breaks the pattern.

Safety invariants:

• Never reorders any segment that already carries a cache_control marker. Explicit intent always wins.
• Never moves text, tool_use, tool_result, or thinking blocks.
• Never touches assistant messages.
• Never mutates the input you passed.

Every time the wrapper actually moves something, it emits an auto-reorder-applied info-level warning so you can see what it did:

new CachedAnthropic({
  apiKey,
  autoReorder: true,
  onWarning: (event) => {
    if (event.code === "auto-reorder-applied") {
      console.info(event.message);
      // → "Auto-reorder canonicalized order-insensitive parts of the request
      //    to preserve the cache prefix: tools alphabetized by name
      //    (3 of 5 moved)"
    }
  },
});

Warnings

The client emits passive warnings (never throws, never blocks a request):

• no-cache-control-found — (Anthropic) you forgot to mark anything cacheable AND auto-cache hasn't activated yet
• cache-write-without-read — your prefix changed call-over-call; cache is broken (carries a diff when diagnoseMisses: true)
• low-hit-rate — rolling hit rate fell below your threshold
• unknown-model — pricing unknown, so dollar accounting is skipped
• auto-placement-applied — info-level (Anthropic): the wrapper just placed cache_control on a newly-stable segment
• auto-reorder-applied — info-level: the wrapper canonicalized order-insensitive parts of the request so the cache prefix would still match
• prompt-too-small-for-cache — (OpenAI, v0.4) the prompt is below OpenAI's 1024-token automatic-cache minimum
• gemini-cache-applied — info-level (Gemini, v0.4): an explicit CachedContent was created or referenced on this call

Route them anywhere:

new CachedAnthropic({
  apiKey,
  onWarning: (event) => logger.warn(event),
});

Roadmap

• ~~v0.2 — auto-placement of cache_control breakpoints based on observed prompt stability~~ ✅ shipped
• ~~v0.3 — safe message and tool reordering to maximize the stable prefix~~ ✅ shipped
• ~~v0.4 — OpenAI and Gemini prompt caching support~~ ✅ shipped
• ~~v0.5 — streaming wrappers; auto-managed Gemini CachedContent lifecycle~~ ✅ shipped
• ~~v0.6 — OpenAI Responses API wrapper; Gemini auto-cache TTL auto-refresh~~ ✅ shipped
• v0.7 — multi-prefix Gemini auto-cache for RAG; Anthropic 1-hour cache TTL support
• v1.0 — persistent stats adapter, middleware mode

Zero runtime dependencies

prompt-cache-optimizer itself has zero runtime deps and uses Node's built-in node:crypto for fingerprinting. The provider SDKs are peer dependencies:

• @anthropic-ai/sdk — required for CachedAnthropic
• openai — optional peer dep, required only if you use CachedOpenAI
• @google/genai — optional peer dep, required only if you use CachedGemini

The OpenAI and Gemini SDKs are dynamically imported on first call — simply importing prompt-cache-optimizer doesn't require either to be installed.

Contributing

PRs welcome — see CONTRIBUTING.md.

Support this project

If this package saved you money on your Anthropic bill, consider buying me a coffee. This project is MIT-licensed and free forever; sponsorship just helps me spend more time on it.

License

MIT © Leonhail Paypa

⭐ If this package saved you money on your Anthropic bill, please star the repo. It's the single biggest signal that helps other developers find it.