@nodellmcache/stream-cache

Streaming response caching for NodeLLMCache. Most LLM apps stream tokens, but a plain cache only stores the final string. StreamCache captures a token stream on a miss (forwarding chunks live as they arrive) and replays it as a stream on a hit — instant first token, zero cost.

Install

npm install @nodellmcache/stream-cache @nodellmcache/memory @nodellmcache/core

Quick start

import OpenAI from 'openai'
import { StreamCache } from '@nodellmcache/stream-cache'
import { MemoryAdapter } from '@nodellmcache/memory'

const openai = new OpenAI()
const cache = new StreamCache<string>({ adapter: new MemoryAdapter<string[]>() })

async function* ask(prompt: string) {
  const completion = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: prompt }],
    stream: true,
  })
  for await (const part of completion) {
    const delta = part.choices[0]?.delta?.content
    if (delta) yield delta
  }
}

// First call streams live from the API and is captured.
for await (const token of cache.stream('Explain Redis', () => ask('Explain Redis'), { provider: 'openai', model: 'gpt-4o' })) {
  process.stdout.write(token)
}

// Second call replays the cached stream — instant, no API call.
for await (const token of cache.stream('Explain Redis', () => ask('Explain Redis'), { provider: 'openai', model: 'gpt-4o' })) {
  process.stdout.write(token)
}

API

| Member | Description | |--------|-------------| | stream(input, generator, opts?) | Async iterable: yields live on miss (capturing), replays on hit | | collect(input, generator, opts?) | Drains the stream to the full chunk[] (also cached) | | invalidate(input, opts?) | Drop a cached stream | | stats() | hits / misses / hitRate / entryCount |

• The generator is any () => AsyncIterable<C> (OpenAI/Anthropic/Vercel AI streams all qualify). C defaults to string but can be any chunk type.
• A stream that throws or is interrupted is never cached — no partial results.
• replayDelayMs adds a delay between replayed chunks for a realistic cadence (default 0, instant).
• cache: false bypasses entirely.

License

MIT