llm-context-budget
v1.0.0
Published
Fit a chat history into an LLM context window. Drops the oldest turns, keeps the system prompt and latest messages, never orphans a tool result, and truncates oversized messages. Works with Anthropic and OpenAI message shapes. Zero dependencies.
Maintainers
Readme
llm-context-budget
Fit a chat history into an LLM context window. Drops the oldest turns, keeps the system prompt and the latest messages, never separates a tool call from its result, and truncates a single oversized message when there is nothing left to drop. Works with Anthropic and OpenAI message shapes. Zero dependencies, typed.
import { fitMessages } from "llm-context-budget";
const { messages, dropped } = fitMessages(history, {
maxTokens: 200_000, // the model's context window
reserve: 8_000, // room for the reply
extraTokens: 1_500, // your system prompt and tool definitions
});
const response = await client.messages.create({ model: "claude-opus-5-5", max_tokens: 8_000, system, messages });Every chat app eventually writes this by hand, and the hand-written version usually drops a tool result without its call, drops the task description, or sends an assistant message first. This package is that function, with tests.
Install
npm install llm-context-budgetNode 18 or newer. ESM and CommonJS, TypeScript types included.
How it decides
- Messages are grouped into units. An assistant message that makes tool calls and the messages carrying its results are one unit. They are kept or dropped together, so the model never sees a
tool_resultwithout itstool_use, or a call with no answer. - Pinned units are never dropped: every
systemmessage, the firstkeepFirstnon-system units, and the lastkeepLastunits. The default pins only the latest turn. - Unpinned units are dropped oldest first until the total fits
maxTokens - reserve - extraTokens. - If the first remaining non-system message is an assistant turn, it is dropped too, since Anthropic requires conversations to start with a user turn. Turn this off with
ensureUserFirst: false. - If the pinned units alone still exceed the budget, the largest message is truncated: middle cut by default, with a marker where text was removed. The original message object is never mutated.
- The result reports what was kept, what was dropped, the final count, and whether it fits.
API
fitMessages(messages, options)
Synchronous. Returns a FitResult.
| Option | Default | Description |
|---|---|---|
| maxTokens | required | Budget for the messages array. |
| reserve | 0 | Tokens held back for the model's reply. |
| extraTokens | 0 | Tokens spent outside messages, such as a top-level system prompt and tool schemas. |
| countTokens | estimateTokens | (message) => number. Plug in an exact counter. |
| keepSystem | true | Never drop role: "system" messages. |
| keepFirst | 0 | Never drop the first n non-system units, for example the original task. |
| keepLast | 1 | Never drop the last n units. |
| truncate | "middle" | "middle", "head", "tail", or "none". |
| truncateMarker | "\n…[truncated]…\n" | Inserted where text was removed. |
| ensureUserFirst | true | Drop a leading assistant turn left behind by trimming. |
interface FitResult {
messages: Message[]; // what to send, original order, original objects unless truncated
dropped: Message[]; // what was removed, original order
tokens: number; // count of `messages`
budget: number; // maxTokens - reserve - extraTokens
truncated: number; // how many messages were shortened
fits: boolean; // false if the pinned set could not be made to fit
}fitMessagesAsync(messages, options)
Same, but countTokens may return a Promise. Use this with a token-counting API.
estimateTokens(message)
The default counter: roughly four characters per token for ASCII, one per character for CJK, plus a small per-message overhead. Good enough for trimming with a sensible reserve. For exact numbers use the model provider's counter.
countMessages(messages, countTokens?), groupMessages(messages), truncateMessage(message, ratio, mode, marker), messageText(message), estimateTextTokens(text)
The building blocks, exported for your own use.
Exact counting with the Anthropic SDK
Counting a message at a time with an API call is slow for long histories. Count once with the estimate, then verify the final result with one real call:
import Anthropic from "@anthropic-ai/sdk";
import { fitMessages } from "llm-context-budget";
const client = new Anthropic();
const model = "claude-opus-5-5";
let { messages } = fitMessages(history, { maxTokens: 180_000, reserve: 16_000 });
const { input_tokens } = await client.messages.countTokens({ model, system, messages });
if (input_tokens > 180_000) {
// The estimate was optimistic. Trim harder using the measured ratio.
const estimated = countMessages(messages);
const scale = input_tokens / estimated;
({ messages } = fitMessages(history, { maxTokens: Math.floor(180_000 / scale), reserve: 16_000 }));
}For per-message exact counts, pass an async counter to fitMessagesAsync:
const result = await fitMessagesAsync(history, {
maxTokens: 180_000,
countTokens: async (m) => (await client.messages.countTokens({ model, messages: [m] })).input_tokens,
});Recipes
Keep the task description and the last three turns:
fitMessages(history, { maxTokens: 32_000, keepFirst: 1, keepLast: 3 });Summarise what was dropped and put the summary back in:
const { messages, dropped } = fitMessages(history, { maxTokens: 100_000, reserve: 4_000 });
if (dropped.length) {
const summary = await summarise(dropped);
messages.splice(1, 0, { role: "user", content: `Earlier in this conversation: ${summary}` }, { role: "assistant", content: "Understood." });
}Fail loudly instead of truncating:
const r = fitMessages(history, { maxTokens: 8_000, truncate: "none" });
if (!r.fits) throw new Error(`Latest message alone is ${r.tokens} tokens, budget is ${r.budget}`);Message shapes
Only role and content are read. Everything else on a message is preserved. Recognised tool patterns:
- OpenAI: an assistant message with
tool_calls, followed byrole: "tool"messages. - Anthropic: an assistant message whose content array has
tool_useblocks, followed by a user message whose content array hastool_resultblocks.
Parallel tool calls are handled: all results following the call belong to its unit.
License
MIT
