@demystify/context — token budget and prompt-cache boundary

Fits context to a per-turn budget, reports everything it dropped and why, and computes where the prompt-cache boundary goes.

Deterministic, pure, zero dependencies, no network, no clock, no model call. Same input, same output — safe in a serverless request path and testable without fixtures.

What it is / when to use it

The cost driver in a production LLM app is not prompt length. It is bloated context and stale history: everything that gets appended "just in case", every turn of a loop that nobody compacts, every retrieved chunk that is no longer relevant. Prompt caching is the biggest lever against that bill — it can take up to 90% off input tokens — but only if something decides where the cacheable prefix ends, and strategic control of that boundary beats caching everything. Caching indiscriminately can cost more and can add latency.

That decision usually lives nowhere. It is implicit in the order a prompt happens to be built in, which is implicit in the order somebody wrote the function. This package makes it explicit, cheap to compute, and testable.

Use it when you are assembling prompts from more than one source, running an agent loop long enough to need compaction, or paying more for input tokens than you expected.

Install

pnpm add @demystify/context     # npm / yarn / bun all fine

Node ≥ 22, ESM. Works in Vercel serverless (nothing here touches disk, sockets, or a clock).

Quickstart

import { createBudget } from "@demystify/context";

const budget = createBudget({
  maxTokens: 128_000,
  reserveForOutput: 4_000,
  minCacheableTokens: 1024, // your provider's floor; below it, don't cache
});

const prompt = budget.assemble([
  { id: "system", text: systemPrompt, stability: "stable", trust: "trusted", pinned: true },
  { id: "tools", text: toolSchemas, stability: "stable", trust: "trusted" },
  { id: "invoice", text: ocrText, stability: "stable", trust: "untrusted", relevance: 0.8 },
  { id: "question", text: userQuestion, stability: "volatile", trust: "untrusted" },
]);

prompt.text;                    // the assembled prompt
prompt.tokens;                  // never more than budget.inputTokens (3398 here)
prompt.breakpoint.index;        // 3 → cache the first three segments
prompt.breakpoint.segmentIds;   // ["system", "tools", "invoice"]
prompt.breakpoint.strandedIds;  // [] → nothing cacheable was left behind
prompt.prefixText;              // send as the cached block
prompt.suffixText;              // fresh input, every turn
prompt.untrustedIds;            // ["invoice", "question"] → route these
prompt.dropped;                 // [] → nothing was lost

Under a tight budget, dropped is the point:

prompt.dropped;
// [{ id: "doc",
//    reason: "over_budget",
//    tokens: 250,
//    detail: "needs 251 tokens, 60 left of the 80-token input budget" }]

Silent truncation is the failure mode this package exists to prevent. Every candidate is either in segments or in dropped — never neither.

The subtle rule: position, not property

A prompt cache is a prefix cache. It matches from the first token forward and stops at the first byte that differs. So cacheability is not a property of a segment, it is a property of a segment's position:

// system (stable) · question (volatile) · tools (stable)
budget.assemble(segments, { order: "as-given" }).breakpoint;
// { index: 1, segmentIds: ["system"], reason: "volatile_segment",
//   endedBy: "question", strandedIds: ["tools"], cacheable: true }

tools is stable. It is byte-identical next turn. And it is billed in full on every single turn, because a volatile segment sits in front of it. Nothing in the code that built that prompt looks wrong.

strandedIds names exactly that: stable segments behind the boundary, which would be free if they were ordered earlier. That is what the default stable-first placement does:

budget.assemble(segments).breakpoint;
// { index: 2, segmentIds: ["system", "tools"], strandedIds: [], … }

Ordering is stable-first by default and preserves your authored order within each class, so the prompt still reads the way you wrote it. Pass { order: "as-given" } when the end-to-end order is semantically load-bearing — and then watch strandedIds.

findCacheBreakpoint is exported on its own and works on any ordered list of { id, tokens, stability }, so you can audit a prompt this package did not assemble.

The pipeline

assemble is select → compress → fit → order → isolate. Each stage is exported (selectSegments, compressSegments, fitSegments, orderSegments, isolateSegments) for hosts building their own.

| Stage | What it does | |---|---| | select | Filters by minRelevance / maxSegments, ranks by pinned → priority → relevance → input position. | | compress | Calls your compressor per segment. This package chooses no compression strategy. | | fit | Admits in rank order until the input cap is spent. Keeps going past an oversized segment so the window still fills. | | order | Places stable-first so the cache prefix is maximal. | | isolate | Reports which placed segments are untrusted so the host can route them. |

Two different orderings, deliberately: rank decides what survives, placement decides where it sits. Emitting a prompt in priority order reshuffles it every turn, which is a cache miss every turn.

pinned: true means the prompt is wrong without it: pinned segments skip the relevance floor and the segment cap, and if one cannot fit, assemble throws rather than handing back a prompt with no system instructions.

trust mirrors @demystify/ai-guardrails' "trusted" | "untrusted" shape without importing it. This package marks; it does not sanitise. Compose the two: guardrails cleans the text, this decides where it goes.

Compaction for agent loops

import { compactHistory } from "@demystify/context";

const compacted = compactHistory(history, {
  maxTokens: 400,
  keepRecent: 4,
  summarise: (turns) => callYourModel(turns), // your function, called at most once
});

compacted.turns;       // [summary, ...the 4 most recent turns, verbatim]
compacted.tokens;      // 229 — fits maxTokens
compacted.summarised;  // 8 turns folded
compacted.dropped;     // every one of them, by index and reason

• The most recent keepRecent turns are kept verbatim.
• The summariser is called at most once, with exactly the turns being folded, in order — and not at all if the history already fits.
• Order of surrender when things do not fit: the summary goes first, then recent turns oldest-first. Recency beats summary in an agent loop.
• If even the last turn exceeds the cap, it is kept and overflow is true. We do not truncate text; half a tool result is a worse input than an honest flag.

Like @demystify/agent-kernel, this package is structurally incapable of calling a model. There is no transport here, and test/no-model.test.ts fails if anyone adds one.

Cost, in integer micro-units

import { cacheSplit, estimateTurnCost } from "@demystify/context";

const cost = estimateTurnCost(cacheSplit(prompt), {
  currency: "USD",
  inputPerMillion: 3_000_000,      // µUSD per 1M tokens
  cachedInputPerMillion: 300_000,
  cacheWritePerMillion: 3_750_000,
});

// For the quickstart prompt above — a 3,390-token prefix, 8 fresh tokens:
cost.uncachedMicro;     // 10194
cost.cacheHitMicro;     //  1041
cost.cacheWriteMicro;   // 12737
cost.savedPerHitMicro;  //  9153
cost.breakEvenHits;     //     1 — hits before the write premium pays back

All amounts are integer micro-units of an explicit currency (µUSD when the currency is USD — the unit @demystify/platform-cache records costMicroUsd in). Never a float. breakEvenHits is null when a boundary can never pay back, which is the concrete form of "caching everything is not the strategy".

No rate card ships with this package. A hardcoded one is stale the week it ships.

Token counting is an ESTIMATE

The bundled default is ceil(code points / 4), the same arithmetic the Demystify gateway uses for estimation. It runs high on CJK and Indic scripts and low on English prose with long common words.

Use it to decide what fits. Do not use it for billing, quota enforcement, or a hard provider limit. Inject a real tokenizer for those:

createBudget({ maxTokens: 128_000, reserveForOutput: 4_000, countTokens: myTokenizer });

countTokens must return a non-negative integer; anything else throws rather than corrupting the arithmetic silently (a NaN count looks exactly like "too big" to a comparison). One more caveat: real tokenizers are not additive — tokens(a + b) can differ from tokens(a) + tokens(b) at the seam. Segment sums are an upper bound in practice, but if you are within a few tokens of a hard limit, measure the rendered text.

What it does NOT do

• It does not call a model. Summarising and compressing are your functions.
• It does not tokenize. No vocabulary is bundled; the default is an estimate.
• It does not talk to a cache. It says where the boundary should go; storing, keying and invalidating is @demystify/platform-cache's job (and tenant-scoped keys are its law, not ours). We hand you prefixText; you hash it how you like.
• It does not sanitise untrusted text. It marks it. @demystify/ai-guardrails redacts and scans.
• It does not truncate text. It drops whole segments and turns, and reports every one. Mid-segment truncation is a compression decision, so it is yours.
• It knows no provider. No model names, no context-window table, no rate card, no vendor cache-breakpoint limits (some providers allow several breakpoints; this computes the one that matters — where the prefix ends).
• It does not rank for you. relevance is a number you supply; there is no embedder here.
• It is not multi-modal. Segments are text. Images and audio have their own token accounting, which belongs with whatever measures them.

API

createBudget(spec): Budget                       // .assemble(segments, options) → Assembly
findCacheBreakpoint(segments, options?): Breakpoint
compactHistory(history, options): CompactionResult
estimateTurnCost(shape, price): TurnCost         // integer micro-units
cacheSplit(assembly): { cachedTokens, freshTokens }
estimateTokens(text): number                     // the documented default heuristic
prepareSegments · selectSegments · compressSegments · fitSegments ·
  orderSegments · isolateSegments                // the pipeline, stage by stage
ContextError                                     // { code, retryable: false }

Drop reasons: over_budget · below_relevance · over_segment_limit · empty. Breakpoint reasons: all_stable · volatile_segment · no_stable_prefix · below_minimum · empty. Error codes: invalid_budget · invalid_segment · invalid_token_count · invalid_price · invalid_history · duplicate_segment_id · pinned_does_not_fit. Every one is a caller bug, so retryable is always false.

Testing

pnpm test                 # 149 tests
pnpm exec vitest --coverage

Tests assert the guarantees, not the implementation: nothing exceeds the cap, every drop is reported with a reason, a stable segment after a volatile one is not cacheable, recent turns survive compaction verbatim, and the same input always produces the same output.

Licence

MIT © Demystify Systems