quiet-cache

⚠️ Work in progress — this does not work yet

Do not use this. It is an in-development prototype, not a working tool. Everything below describes the intended design; much of it is unverified and some of it is not wired up at all.

Known state, as of 0.5.0-alpha.3:

• Published as @quietops/quiet-cache. The unscoped name quiet-cache is not ours; make sure you have the scope.
• Never run end-to-end against the live API. The proxy has only been exercised against local fixtures. Expect it to fail on real traffic.
• The test suite has not been run against this revision. Treat every passing-looking claim in this README as unconfirmed.
• The compression and breakpoint heuristics are unvalidated. The savings figures shown below are illustrative, not measured. Stage 1 is lossy and has not been evaluated for how often it degrades an agent run.
• The hosted metrics collector is not live, so opting in to telemetry does nothing.

Pointing a real workload at this means routing your API traffic through untested code. Wait for a tagged release.

A local LLM proxy that cuts what you pay for the Anthropic Messages API.

Agent loops are pathologically repetitive. The same system prompt and the same dozen tool schemas are re-sent on every single turn. The same file gets pasted into the transcript three times. A directory listing from forty turns ago is still being re-billed as input on turn forty-one. You pay for all of it, again, every call.

quiet-cache sits on loopback between your agent and the API and removes that waste on the way past:

| | Stage | Lossless? | What it does | |---|---|---|---| | 1 | Compress | no | Elides stale tool output and repeated blocks from the transcript | | 2 | Breakpoints | yes | Auto-places cache_control markers on the stable prefix | | 3 | Replay | yes | Serves byte-identical repeat calls from disk, unbilled |

Stage 2 is usually where the money is. Prompt caching reads a marked prefix at 10% of the base input rate, and almost nobody places the markers by hand — so the proxy does it, from traffic it can already see.

No SDK changes. No account. No dependencies.

Intended output (illustrative — these numbers are made up, not measured):

$ quiet-cache report
since:              2026-07-31 09:12
requests:           4187

billed input:       2.41M
billed output:      918.0k
prefix writes:      412.7k  (cache creation, billed at a premium)

avoided — replays:  1.02M  (served from disk, nothing billed)
avoided — prefixes: 8.73M  (prompt-cache reads, 90% of base rate)
avoided — trimmed:  1.44M  (estimated; removed before sending)
avoided — total:    11.19M

reduction:          77.1%

Install

npm install -g @quietops/quiet-cache

Pin the version while the package is pre-release:

npm install -g @quietops/[email protected]

Or clone and link it to work on it locally:

git clone https://github.com/quiet-cache/quiet-cache
cd quiet-cache && npm link

Node 18+. Zero runtime dependencies.

Use it with Claude Code

quiet-cache on        # writes ANTHROPIC_BASE_URL=http://127.0.0.1:8787
quiet-cache serve     # start the proxy, leave it running

quiet-cache off restores whatever ANTHROPIC_BASE_URL was there before. The displaced value is saved into ~/.claude/settings.json under quietCache.previousBaseUrl, and a one-time backup of the whole file is kept at settings.json.quiet-cache.bak.

Verify where your traffic is actually going, any time:

$ quiet-cache doctor
  ok   cache dir writable
  ok   base URL is loopback or unset
  ok   proxy reachable

That middle check exists because ANTHROPIC_BASE_URL is the single most security-relevant setting on the machine. Anything holding it that isn't loopback is reading every prompt you send.

Use it as a library

If you're calling the SDK directly and don't want another process in the loop, apply the same pipeline inline:

const Anthropic = require('@anthropic-ai/sdk');
const { optimize } = require('quiet-cache');

const client = new Anthropic();

const { body, stats } = optimize(requestBody);
// stats -> { trimmedTokens: 3184, breakpoints: 3, prefixTokens: 21903, applied: [...] }

const res = await client.messages.create(body);

Or wrap the call to add disk replay on top:

const { memoize } = require('quiet-cache');
const create = memoize((body) => client.messages.create(body));

// First call hits the API. Every identical call after that is free.
await create({
  model: 'claude-opus-5',
  max_tokens: 1024,
  messages: [{ role: 'user', content: 'Summarise RFC 9110 in three bullets.' }],
});

Lower-level pieces are all exported: compress, injectCacheControl, Store, Ledger, requestKey, estimateRequest, createServer.

Stage 1 — context compression

This stage is lossy. It changes what the model sees. It is on by default because the default transforms are conservative, but there is no setting under which it is free. Three invariants bound the damage:

1. system is never touched.
2. The last 4 messages are never touched — that's the live part of the conversation the model is actually reasoning about.
3. Nothing is deleted silently. Every elision leaves a [quiet-cache] marker so the model can tell something was removed and re-run the tool if it needs it.

| Transform | Default | What it removes | |---|---|---| | toolResultWindow | 8 messages | Payloads of tool_result blocks older than the window. The single largest source of dead weight in agent transcripts. The block itself stays, so tool_use_id pairing survives. | | dedupe | on, ≥512 chars | Later copies of an identical large text block, replaced with a back-reference. | | maxBlockChars | off | Middle-elides oversized text blocks, keeping both ends. | | maxMessages | off | Drops whole message pairs from the middle. Most destructive; off by default. Never splits a tool_use from its tool_result. |

quiet-cache serve --tool-window 4 --max-block 8000 --max-messages 40
quiet-cache serve --no-compress          # stage 1 off entirely

Stage 2 — automatic prompt-cache breakpoints

Lossless. cache_control markers change what the API bills, not what the model sees.

The API allows four ephemeral breakpoints per request, and they're cumulative prefixes — so placement order matters, and marking a volatile boundary is worse than not marking it, because every miss pays the write premium for nothing. quiet-cache places them earliest-and-most-stable first:

1. End of tools — longest-lived thing in the request, often the largest.
2. End of system — a bare string is normalised into a block array so it can carry a marker at all.
3. Message-history boundaries — walking backwards from the second-to-last turn, so markers land on content that has already settled.

Skipped entirely when the request is under ~1024 estimated tokens (below the cacheable floor), or when you've placed your own markers — your placement wins.

quiet-cache serve --no-breakpoints

Stage 3 — disk replay

Byte-identical repeat calls are served from a local content-addressed store and never reach the API at all.

The cache key is a SHA-256 over a canonical serialization of exactly these fields: model, system, messages, temperature, top_p, top_k, max_tokens, stop_sequences, tools, tool_choice, thinking. Canonicalization sorts object keys recursively, so requests differing only in JSON property order collide as they should. Anything not in that list — metadata, request ids, the stream flag — is ignored, because it can't change what the model returns.

Two ordering details that matter:

• The key is computed after compression, so requests differing only in material compression would have removed still collide.
• It's computed before breakpoint injection, because cache_control affects billing and not output. Hashing it would fragment the cache for nothing.

Keys are namespaced by a 12-character hash of the API key, so two accounts on one machine never read each other's entries.

What is and isn't touched

| Request | Behaviour | |---|---| | POST /v1/messages, non-streaming | All three stages | | POST /v1/messages, stream: true | Stages 1 and 2. Never replayed — reconstructing SSE timing from a buffered body isn't worth the fidelity loss | | Non-200 responses | Never stored. A cached 429 would pin a transient failure for the whole TTL | | Every other path | Transparent passthrough, untouched |

Credentials are forwarded upstream unchanged and never written to disk. Only a truncated hash of the key is stored, and only as a namespace prefix.

Storage

~/.cache/quiet-cache/
├── entries/
│   ├── 0a/1f9c….json
│   └── 0b/77e2….json
├── stats.json
└── ledger.json

Entries are sharded on the first byte of the key. Writes go to a temp file and are rename(2)d into place, so a crash mid-write can't leave a torn entry for a concurrent reader.

Two independent limits, both enforced on write:

• TTL — default 7 days. Expired entries are dropped on read and by purge.
• Size cap — default 512 MiB. When exceeded, least-recently-read entries are deleted until the store is back under 90% of the cap.

quiet-cache serve --ttl 1 --max-size 128
quiet-cache purge          # drop expired
quiet-cache purge --all    # drop everything

Accounting

quiet-cache report tracks four channels separately, because they aren't interchangeable and averaging them produces a flattering, meaningless number:

• replays — served from disk. Nothing was billed.
• prefix hits — prompt-cache reads, counted at 90% of the tokens since they're billed at 10% of base.
• trimmed — input tokens removed by stage 1. Estimated, not measured.
• billed — what actually went on the invoice.

Every figure except trimmed comes from the usage block the API itself returned. Token estimates (~4 chars/token) are only ever used internally to gate heuristics — never presented as if they were measured.

quiet-cache report
quiet-cache report --reset

Configuration

| Variable | Default | Meaning | |---|---|---| | QUIET_CACHE_DIR | ~/.cache/quiet-cache | Cache root | | QUIET_CACHE_UPSTREAM | https://api.anthropic.com | Where misses are forwarded | | QUIET_CACHE_TELEMETRY | unset (off) | Set to 1 to opt in to anonymous metrics | | CLAUDE_CONFIG_DIR | ~/.claude | Where settings.json lives |

QUIET_CACHE_UPSTREAM exists for teams already behind an API gateway — point it at the gateway and quiet-cache layers underneath.

Telemetry is off unless you set QUIET_CACHE_TELEMETRY=1, and it covers counters only — hit ratio, breakpoint placement rate, tokens avoided. No prompt content, no message bodies, no file paths, no credentials. It's the same object you can read yourself at /__quiet-cache/health. The hosted collector isn't live yet, so right now opting in does nothing but print a notice on startup.

Response headers

Every response is annotated so you can see what the proxy did:

x-quiet-cache: HIT | MISS | STREAM
x-quiet-cache-key: 0a1f9c3d…
x-quiet-cache-age: 412
x-quiet-cache-trimmed: 3184
x-quiet-cache-breakpoints: 3

GET http://127.0.0.1:8787/__quiet-cache/health returns cache stats and the token ledger as JSON if you'd rather scrape it.

CLI reference

quiet-cache serve [options]     Run the proxy
quiet-cache on   [--port N]     Point Claude Code at the proxy
quiet-cache off                 Restore the previous ANTHROPIC_BASE_URL
quiet-cache status              Show config + whether the proxy is up
quiet-cache report [--reset]    Tokens billed vs. tokens avoided
quiet-cache stats               Disk cache size and hit rate
quiet-cache purge [--all]       Drop expired entries (or everything)
quiet-cache doctor              Check that the setup is sane

Caveats

• Stage 1 is lossy. An agent that needed the tool output you elided will re-run the tool, and you'll pay for the turn that discovers this. If your workload can't tolerate that, run --no-compress and keep stages 2 and 3.
• Non-determinism is on you. At temperature > 0 a replayed response is one sample, served repeatedly. Use a short --ttl if you want fresh samples.
• Breakpoints cost money when they miss. A cache write is billed at a premium. On short-lived, highly variable prompts the placement heuristic can lose; quiet-cache report shows prefix writes separately so you can tell.
• Tool-use loops cache per-turn, not per-conversation. Usually what you want, but a partially-replayed loop can interleave old and new turns.

Tests

npm test

License

MIT