@broberg/chat

The fleet's AI-chat core. A conversation loop with a tool registry, streaming typed frames — framework-free, storage-free, and with zero dependencies.

npm i @broberg/chat

The line this package exists to make unwritable

The cms session measured this in their own 64-tool chat:

tools.filter(t => !t.permission || hasPermission(user, t.permission))

!t.permission || — a tool that declared no permission passed. 60 of their 64 declared none, so a read-only user was handed 61 tools, 30 of them mutating.

It reads exactly like a permission check. It is one, for the four tools that declared something. For the rest the default pointed the wrong way.

So permission is required, enforced twice — the type rejects a literal without it, and defineTool() throws, because a registry built at runtime has no compiler.

import { defineTool, createChat } from "@broberg/chat";

const lookup = defineTool({
  name: "roster_lookup",
  description: "Look up one employee across roster, users and audit-log",
  permission: "roster.read",          // required. no default. no fallback.
  parameters: { type: "object", properties: { email: { type: "string" } } },
  run: (args, ctx) => ctx.api.get(`/admin/roster/${args.email}`),
});

A tool that can act must not decide whether it may

sanne's rule, and the reason acting is safe at all. Their book_appointment calls an endpoint that answers consent_required — proven with raw calls bypassing the UI.

The core enforces the structural half: a tool's run() receives only the ctx you passed in. It is never handed a database, an engine or a client, so your routes stay the authorization boundary. That is precisely why cms's defect could exist — their tools called the engine directly and skipped every HTTP permission gate.

A permission must not be born in the chat layer

cms measured this on 2026-08-28, and it is the sibling of the rule above.

Their owner decided a viewer must not see form submissions. The obvious change is to deny the three forms.read tools — one permission, three tools, done. So they checked what else read that data, and found four doors the chat never touched:

| door | what it actually asked | |---|---| | the submissions list | is someone logged in | | a single submission | nothing at all — it relied on a proxy guarding /api/admin/*, which answers authenticated, not permitted | | CSV export | is someone logged in — and that is the whole dataset in one file, not a preview | | the /admin/forms page | whether the FEATURE was enabled for the tenant, never whether this person may see it |

Had they changed only the tool permission, the chat would have refused while the page beside it served the same names, and the export served all of them at once.

A permission enforced only in the chat is not a permission. It is a chat setting.

can(permission, caller) should therefore be a reader of authorization your app already has, never the place it is first decided. If a permission string exists only because a ChatTool declared it, every other route in your product is still open — and the chat's refusal will make it look closed.

Before you register a tool, ask what else reaches the same data. Every answer that is not "the same permission" is a door.

The pattern behind the three questions

The same owner was asked three visibility questions in one day, and answered all three the same way — a reader may read what is published, never the record behind it:

| may a viewer see… | answer | |---|---| | form submissions | no | | deleted content (trash) | no | | old versions (revisions) | no |

They look like three features. They are one rule about the data, and it is worth deriving rather than deciding tool by tool: published is what a reader was given; a submission, a deletion and a superseded draft are all the record of how it got there, and none of them was written for that audience.

So when a tool exposes a history, a bin, or anything a person submitted, the default is no — and the question to put to the owner is the one about the data, not the one about the tool. Ask "may a reader see what other people submitted?" and you get an answer in two seconds. Ask "should list_form_submissions require forms.read?" and you get a shrug, because it is not a question anyone outside the code can answer.

Permission is asked per caller, not matched against a list

const chat = createChat({
  model,                               // injected — see below
  tools: [lookup, invite],
  can: async (permission, caller) =>
    !caller.accessRevokedAt && (await grants(caller)).includes(permission),
});

can is required whenever tools are registered. There is no permissive default — "everyone may use everything" is the same mistake as !t.permission || moved one level up: it looks like configuration and behaves like an open door.

Async because a real answer is a lookup, not a list. fd-sundhed measured why: their role is not the gate on its own — access_revoked_at sits beside it, and an admin whose access had been revoked walked in until a guard checked both.

A denied tool is invisible, not refused. It is never offered to the model, so it cannot be proposed — and the name is refused again at execution if it arrives anyway. Two gates, because in cms's case the filter was the only one.

The model is injected, never imported

const model: ModelFn = async function* ({ system, messages, tools }) {
  // call @broberg/ai-sdk here — the fleet chokepoint for cost + provider policy
};

Two reasons, and the second was bought the day this shipped:

1. The core is testable against a fake model — a full tool round with no key and no network.
2. It carries no version pin. @broberg/logger promised it "cannot leak a secret" while pinned to a secret-scan four minors stale, because a caret on 0.x locks the minor. As buddy put it: a package cannot know its promise has become untrue because of something underneath it. This one has nothing underneath it.

Streaming from the first version

for await (const frame of chat.run({ messages, caller, ctx })) {
  // "text" · "tool-call" · "tool-result" · "error" · "done"
}

Not an enhancement. trail measured 13.1 seconds average response — on the fastest tier. A non-streaming core would have to be rewritten rather than extended.

done carries a reason: complete, or max-rounds when the loop was cut off mid-work. A caller that cannot tell those apart reports a truncated answer as a finished one.

A broken tool degrades the answer, never the conversation. A tool that throws or rejects produces an error frame the model can see and recover from, and the stream still reaches done.

"I cannot look that up" is not "no"

The sharpest failure in the whole survey, and the reason the core owns a small prompt fragment:

Christian asked Eir whether Sanne sells anything. Eir said no — confidently — because the shop tool was missing.

The model was not confused. It was blind and sounded certain, and a missing capability became a false statement about a business. corePrompt() makes the honest answer reachable; everything else in the prompt is yours.

The bot is called Aidan

One value, one place. Override it for a whole site with a single environment variable:

CHAT_BOT_NAME=Eir

A test asserts the literal appears in exactly one source file — a name repeated across files is a name that drifts the first time one copy is edited.

Mounting it — Stack A and Stack B

// app/api/admin/chat/route.ts          — Next.js App Router
import { createChatRoute } from "@broberg/chat/next";

export const runtime = "nodejs";
export const dynamic = "force-dynamic";   // never cached, never prerendered

export const POST = createChatRoute({
  chat,                                              // createChat(), model injected
  getCaller: async (req) => await readProfile(req),  // YOUR existing pattern
  getCtx:    async (req, caller) => ({ api: apiFor(caller) }),
});

// Hono (Bun/edge)
import { chatHandler } from "@broberg/chat/hono";
app.post("/api/chat", chatHandler({ chat, getCaller, getCtx }));

// the browser — framework-free, no dependency
const res = await fetch("/api/admin/chat", { method: "POST", body: JSON.stringify({ messages }), signal });
for await (const frame of readChatStream(res)) { … }

Both adapters are the same code over web-standard Request/Response, driven by one shared table of test cases. Fix one half of a pair is a measured fleet defect, so a behaviour that holds in Next and not in Hono is a red test rather than a production discovery. @broberg/chat/http is that shared half, exported for anything else — Bun.serve, Workers, Deno.

Still no dependency, still no version pin. No subpath imports @broberg/ai-sdk; the model stays injected. A caret on 0.x locks the MINOR, so a subpath that depended on another @broberg package would hand consumers a version they never chose — which is exactly how @broberg/logger shipped a promise it no longer kept.

getCaller is required, and null is a 401

Not an empty tool list. An unauthenticated request that still reaches the model is still an LLM bill, on a surface where the stranger decides the volume — and on an internal admin chat it is simply the wrong answer. A test asserts the model records zero calls.

The caller is resolved server-side, per request, and can never arrive in the request body. Every message is rebuilt from the three fields we know, so a role or permission smuggled onto one is dropped rather than carried.

Knowledge comes from Trail, as a tool

import { trailRetriever } from "@broberg/chat/trail";

const knowledge = trailRetriever({
  baseUrl: "https://app.trailmem.com",   // ← see the two-host trap below
  kbId: "fd-sundhed-admin",              // CONFIG. Never an argument the model supplies.
  tenant: "fd-sundhed",                  // required on the app route
  token: process.env.TRAIL_API_KEY!,
  permission: "knowledge.read",
  fetch,                                 // injected — every state below is testable with no network
});

⚠️ Two hosts, and one key does not fit both

Measured by trail on a live call — and they fell into it themselves while answering us:

| | | |---|---| | app.trailmem.com | the admin proxy. App key + X-Trail-Tenant. It resolves the tenant and forwards with the tenant's own bearer. Use this unless someone handed you a tenant key. | | engine.trailmem.com | wants the tenant key directly. sanne call this one because they were given that key. |

The same key returns 200 on the first and 401 "Invalid or revoked API key" on the second. The error blames the key. That is why unauthorized is its own reason here rather than a generic http_error — otherwise you rotate a credential that was never the problem.

Christian, 2026-08-27: "ALLE CHATS SKAL anvende trail — det er IKKE til diskussion." So this is the fleet's one knowledge path, which means a defect here is a defect in every chat at once. Three properties follow from that.

1. Three outcomes, never two

{ status: "hit",         passages, freshness, truncated? }
{ status: "empty",       freshness, note }
{ status: "unavailable", reason, note }     // ← no passages field at all

A typed result, not a string, and this is the whole point. sanne's tool does distinguish four failures — and their prompt merges them again: "# Hvis trail_retrieve returnerer ingenting eller fejler", one branch, telling the model to answer from its own general training knowledge and never to say it cannot answer. So when Trail is down, a zone-therapy clinic's assistant answers health questions from generic training knowledge, in the practitioner's voice, and nobody can see the knowledge base was never asked.

That is the Eir shop incident one storey down. A prompt cannot merge two states it receives as different values — so unavailable is a different value, and its instruction is the opposite one: do not answer from your own general knowledge.

Errors never travel in the content channel either: unavailable carries a short machine reason, never the provider's body. A model handed [error] HTTP 500 … reads an error message as knowledge.

2. The knowledge base is configuration, never an argument

kbId and tenant do not appear in the schema the model sees, and either arriving in the model's arguments is ignored. If they were arguments, a model could be talked into another tenant's knowledge — and fd-sundhed alone is getting two knowledge bases, written for readers with different rights.

And this lock is currently the only one. trail measured that an app key can be scoped to several tenants, with X-Trail-Tenant choosing between them. A partner scope bound to a single knowledge base is carded at trail (their F205.1) and not built — so until it is, the configuration here is the real barrier, not theirs.

3. Freshness — the oldest date, and what is undated counted separately

Trail ships updatedAt per passage (their F213.1, live 2026-08-27): ISO-8601 UTC with Z, or nothing. Two decisions in how we use it:

The OLDEST date, not the newest. An answer is only as current as its stalest source — if it rests on three passages and one is from April, part of the answer is from April. Reporting the newest would flatter it.

unknown is counted separately, never folded into the date. Some passages may carry a date and others none; saying only "as of June" would hide that the rest is undated. Three states, not two:

{ oldestUpdatedAt: "2026-04-16T16:31:49.278Z", unknown: 0, note: "…last updated … at the oldest" }
{ oldestUpdatedAt: "2026-06-01T10:00:00.000Z", unknown: 2, note: "…and 2 passage(s) carry no date at all" }
{ oldestUpdatedAt: null,                       unknown: 2, note: "None of this knowledge carries a date" }

Only ISO-8601 UTC with Z is accepted; anything else is UNKNOWN. Trail's own column holds 2026-06-22 12:07:09 beside 2026-04-16T16:31:49.278Z — both UTC, only one saying so — and they normalise server-side. Should that ever regress, a bare 2026-06-22 12:07:09 reaching us would be read by Date as local time and become a confidently wrong date two hours out. Refusing it degrades to "unknown" instead: a wrong date is worse than none.

⚠️ If you build on this, pin a non-UTC timezone in your test — and assert the pin took effect. bun test defaults to TZ=UTC and CI containers are UTC, and under UTC local time and UTC coincide, so this entire class of defect does not exist there. trail's own first test was green against the naive implementation for exactly that reason. Test summer and winter, or +1 and +2 both pass on a single offset.

There is also our own ceiling on top of Trail's maxChars/topK, and what it dropped is reported. sanne cap nothing client-side, so the day Trail stops honouring maxChars they have none at all.

Not live-verified. Every state above is proven against a recorded response measured from production, with fetch injected. No call has been made against a real Trail from this package — a Trail tenant exists only after a human Google login, so the first live run waits on that.

Set your own timeout, and mean the unavailable branch. trail have no rate limit on retrieve, and they volunteered why the third state matters: their engine was down for ~75 minutes the evening this shipped, 502 to its own health endpoint — a full volume, no successful deploy since 23 August, and a process that had not restarted since 4 July, so nothing ever noticed. Their words: our uptime does not carry the assumption that we always answer.

Two boundaries worth knowing before you ship

0. An assistant turn that called a tool carries the call. ChatMessage has toolCalls?: { id, name, args }[], set by the loop on the assistant turn that asked — even when the model said nothing before calling, which is the case that took cms down. Your adapter reads it to build the provider's tool_calls array. Ignore it and a chat without tools is unaffected.

1. The transcript is client-supplied. A caller can forge their own history, including a tool message. Anything that matters must come from a tool call in this turn, never from history — the same rule as a tool that can act must not decide whether it may.

2. A tool RESULT is passed through verbatim, secrets and all. Not an oversight: it is the consumer's data on their own surface, and quietly rewriting it would be a worse surprise than passing it on. If a tool can return a credential, either do not return it or wait for redaction (F079.6). A test pins this boundary so nobody can believe otherwise.

What the adapter itself puts on the wire is guaranteed: no ctx, no permission string, no stack trace. Asserted by scanning the emitted bytes.

An overflowing conversation must not become a dead one

const chat = createChat({ model, history: "standard" });   // a named profile

…or the full object, when you want the numbers yourself:

const chat = createChat({
  model,
  history: {
    strategy: "window",        // or "compact" — REQUIRED, there is no default
    maxInputTokens: 120_000,   // REQUIRED, and declared by YOU (see below)
    keepRecent: 6,
    // strategy: "compact" also needs:
    // summarise: async (older) => (await ai.chat({ prompt: `Summarise: …` })).text,
  },
});

It is one product question, not two numeric fields

{ strategy, maxInputTokens } asks in a unit nobody decides in. The question a person actually answers is:

What happens when the conversation gets too long — and how long may it get?

That has a different answer for someone authoring content for hours than for a visitor asking three questions. cms put it to Christian in these words and he answered in two seconds; it could not have been asked in tokens. So the translation lives here, once, instead of in every consumer:

| you say | what happens when it runs long | what it costs | |---|---|---| | "visitor-qa" | forget the oldest — a stranger asks a handful of questions and leaves | free and instant | | "standard" | forget the oldest | free and instant, but see the warning below | | "long-authoring" | summarise the oldest, so the thread survives | one extra model call each time it fires — you supply summarise | | (omit history) | no limit — a choice, not the absence of one | the dead conversation above. See below. |

⚠️ "Forget the oldest" usually drops the user's opening instruction. Tone, language, role, "answer in Danish, and always mention the free consultation" — that is turn one, so it is the first thing to go. The conversation then carries on without it and everything looks entirely normal. If the opening instruction matters, either put it in your systemPrompt (which is never dropped) or use "long-authoring" so it is summarised rather than lost.

⚠️ Omitting history is "no limit", and it is a choice with a name. It is what every consumer has today if they skip the field, and the consequence is the whole reason this module exists: the conversation does not get expensive, it dies, and a retry resends the same oversized payload for ever.

The numbers behind the profiles (HISTORY_PROFILES) are ours, derived and not measured — chosen to sit comfortably inside a 128k context even with a large tool set. They are a safe floor to start from, not the most your model can take. Steer one without leaving it:

import { resolveHistoryProfile } from "@broberg/chat/history";
const config = resolveHistoryProfile("long-authoring", { summarise, maxInputTokens: 96_000 });

An unknown profile name throws, naming the valid ones. It never falls back to a default — that would be us making a silent decision about your bill, and about which of your user's turns survive, on the strength of a typo.

The defect this exists for, measured by cms in their own chat: nothing truncates, so the client sends the whole conversation every message; the route's maxTokens is the output limit; the provider 400s and the raw error reaches the user. And the part that makes it serious:

Because nothing truncates, a retry resends the same oversized payload. The conversation is not expensive, it is dead — from the moment it tips, after a long session, which is exactly when there is most to lose.

So the test that matters is not "the message got shorter". It is the next turn on the same conversation succeeds.

Choosing maxInputTokens — three things that decide it, and none is the model's spec sheet

We do not have a measured number for you, and the field is required precisely because there is no safe default. But three things decide yours, and a consumer should not have to discover them one at a time:

1. The ceiling is not the model's context window. It is the window MINUS your system prompt, MINUS your tool schemas, MINUS room for the answer. The tool schemas are the part people forget, and they are sent on every call.

Measured by cms, 2026-08-28, over their 64 tools: names 967 chars, descriptions 8,543, input schemas 18,756 — 28,266 characters: ≈7,100 tokens at our default rate, ≈8,300 at the Danish rate they measured (below), before the conversation starts. Two thirds of it is JSON schema, not prose. And it grows every time somebody adds a tool.

Since 0.4.0 we count it for you. createChat hands prepareHistory the tools this caller may actually use, so the number compared against your limit is what goes on the wire — a tool added tomorrow is counted tomorrow, and a caller denied a tool is not charged for it. There is nothing to keep in step.

Until 0.4.0 the guard counted only messages and the system prompt, so for any consumer with a real tool set it was low by the whole cost of the schemas — and low is the green direction: it reported room while the provider was already over. Reported by cms the day they went to production on 0.3.0. The same measurement found their own prompt-size alarm watching the system prompt and not the tool schemas, under-reporting the fixed cost the same way.

Calling prepareHistory yourself? Pass your tools as the fourth argument. And if something else fixed rides along on every call — a gateway preamble — declare it as fixedOverheadTokens; a value that is not a number is refused rather than counted as zero.

If the tool schemas alone exceed your limit, you get overhead_exceeds_limit rather than cannot_reduce. They are different problems with different fixes: one says shorten the conversation, the other says offer this caller fewer tools, or raise the limit. Merged into one state, it would send you to trim a message that was never the problem.

2. estimateTokens is ~4 chars/token, and that is an ENGLISH rule of thumb. Danish (æ ø å, longer word forms) costs more tokens per character, so the default estimate under-counts — the dangerous direction, because you believe you have room you do not have.

Measured by cms on their own Danish prose, 2026-08-28: 3.41 characters per token. That is ~17 % more tokens than our default assumes, in exactly the direction we warn about. It is recorded here as evidence that you should inject your own estimator — not as a new default. One consumer's corpus is not the fleet's, and a second number pretending to be universal would be worth less than the measurement that shows why to take your own. estimateTokens is injectable for exactly this reason: pass your provider's real tokenizer, or set the ceiling low enough that the estimate's error cannot reach it. Write down which of the two you chose.

3. The two errors are not symmetric. Too low costs some premature compaction — you lose a little context. Too high brings back the dead conversation this whole module exists to prevent. Let that decide how conservative you are.

The method, because it is a measurement and not a lookup: send increasing payloads with your REAL system prompt and REAL tool schemas until the provider 400s. Set the limit 20 % below that. Record both numbers with the date — a provider ceiling is not a constant, and an undated figure looks like a measurement a year later.

Compaction changes what the MODEL sees, never what the USER can read

This module never mutates the array you give it, and neither does the loop. Whatever you persist stays complete and verbatim, however much was left out of the payload. cms's rule, from a real user they are not: somebody uses their admin chat as a working tool and may expect to re-read a session word for word.

Three outcomes, and a failure is one of them

{ status: "unchanged", messages, estimatedTokens, warning? }
{ status: "reduced",   messages, estimatedTokens, dropped, strategy }
{ status: "failed",    reason: "compaction_failed" | "cannot_reduce" | "overhead_exceeds_limit", messages, note }

compaction_failed (your summariser threw), cannot_reduce (there was nothing left to remove) and overhead_exceeds_limit (the tool schemas and fixed prompt do not fit on their own) are never the same value, and a failure returns the transcript unchanged — never half-shortened. sanne's rule, generalised: when a layer beneath the chat can fail, the failure carries its own state all the way up and never merges with "nothing found".

⚠️ note is written for a developer. Never show it to your user.

Every history frame carries a note — an English sentence describing what happened, for someone reading a log. It is not for the person whose conversation was just shortened.

cms passed it straight through to an end user (2026-08-28): an English sentence about a mechanism turned up mid-conversation with a Danish customer. They were not being careless — until 0.6.0 there was nothing else on the frame to act on, so the prose was the only signal. That was our defect, not theirs.

Switch on the codes and write your own sentence:

if (f.type === "history") {
  log.info(f.note);                                  // ours, English, for you
  if (f.action === "reduced")
    tell(f.strategy === "compact"
      ? "Jeg har skrevet de ældste beskeder sammen for at gøre plads."
      : "De ældste beskeder er ikke længere med. Gentag gerne det vigtigste.");
  if (f.action === "failed")
    tell(f.reason === "overhead_exceeds_limit"
      ? "Der er for mange værktøjer slået til. Kontakt din administrator."
      : "Den sidste besked er for lang. Prøv at dele den op.");
}

reason is on every failed frame and strategy on every reduced one — the same three states the module keeps apart internally, now reaching the boundary where somebody needs them. We ship no user-facing prose: we do not know your language, tone or audience, and a sentence we invented would need translating anyway. The code is ours; the words are yours.

⚠️ window losing the opening instruction is MEASURED, not predicted

Run live by cms against Mistral in their production container, 2026-08-28, with a marker planted in the first message that the final answer could not produce without it:

--- window ---   maxInputTokens lowered until the ceiling was actually reached
history frames : reduced(3), reduced(5), reduced(7), reduced(9), reduced(12)
error frames   : 0
marker         : GONE
last answer    : «Jeg kan ikke opsummere noget, medmindre du giver mig noget at opsummere.»

It is worse than this README used to say. The model had forgotten the whole conversation and said so politely, with zero errors. A user does not read that as "the system dropped my context" — she reads it as "this assistant is unreliable".

| | | |---|---| | window loses the opening instruction | measured — mistral-small, 5 reductions, 0 errors | | compact preserves it | measured, WITH A CAVEAT — see below | | cannot_reduce as its own state | measured useful — it told them the problem was not the old messages |

The caveat is load-bearing, so it is written as they measured it: compact preserved the instruction in their run, with a summariser written to preserve it. Not "compact preserves it". Three of four summaries named the instruction explicitly; the fourth did not, and the marker survived anyway — probably because keepRecent held the newest turns. So preservation is likely, not guaranteed, per summary.

That distinction is not pedantry. The next consumer writes their own summarise, and with the naive version the same team measured the opposite:

⚠️ Your summarise prompt: a system instruction loses to a question standing last

cms's first version passed the excerpt as the prompt and the instruction in system. The model saw a conversation ending in a user question and did the obvious thing — it answered it, in the assistant's own voice:

[summarise] RETURNED: "En typisk misforståelse om zoneterapi er, at det kan
                       helbrede alvorlige sygdomme … \n\nBananflue"

Those answers were then inserted as the record of what had been said. A long session would have had its real history replaced by an invented continuation, with the opening instruction lost inside it — and zero error frames. The conversation carried on from a past that never happened.

Before and after, same setup, real provider:

| | | |---|---| | naive prompt | 4 reductions → marker GONE, last answer about something else | | fenced prompt | 6 reductions → marker still there |

What fixed it — theirs, and the part to copy: fence the excerpt as unmistakable data, label each turn, put the instruction beside the data rather than in system, say explicitly "do NOT answer the questions in the excerpt", and make the last thing the model reads "Summary:" — not the user's question.

We cannot write that prompt for you (only you know what matters in your conversations). We can tell you the shape that fails.

Their first run never reached the ceiling, and the probe said "PROVES NOTHING" of its own accord rather than passing. If you run this yourself, build that in — a probe that checks only "no errors, text came back" is green on exactly the loss it exists to find.

The loop surfaces all of it as typed history frames (warned · reduced · failed), and an unshrinkable turn ends with done: "too-large" without calling the model — sending a payload you know is too large is how the conversation dies.

⚠️ You declare the limit. It is not read from the model registry.

Measured by cms and confirmed by ai-sdk: a model object carries exactly [id, alias, provider, available, status, note, source] — there is no context window in it, on any of the ten models. The only number that looks like one is maxTokens, which is a per-call output limit. Read it as "the window" and you get a number for something else entirely, which is worse than no number because it looks like an answer.

estimateTokens is a rough heuristic (~4 characters per token) and says so. Inject your own estimate if you have a real tokenizer.

What is deliberately not here

RAG over history — store everything, retrieve only what is relevant. It is last by Christian's order ("vi skal have noget i drift med FD Sundhed FØR vi har RAG klar"), and it is not in the HistoryStrategy union rather than present-and-throwing: an option that exists and throws fails in production; one that does not exist fails in your editor.

Prompt-caching is not here either, and needs nothing from you: @broberg/ai-sdk 0.31.0 turned it on by default with a content-derived key. Measured live: $0.004411 → $0.000458.

A ceiling, and a wall in front of a public endpoint

A public chat is an open LLM-spend faucet on a surface where strangers decide the volume. An internal admin chat is not, which is why none of this is on by default — and why mode: "public" is the stricter setting, not the relaxed one.

The number the ceiling reads

The first thing measured when this was built: the core had no cost channel at all. ModelEvent yielded text and tool-call and nothing else, so the figure a cap reads was discarded one layer below the guard that would read it. usage was added to that union — additively, so a ModelFn that never yields it keeps working:

const model: ModelFn = async function* (req) {
  const res = await ai.chat({ tier: "smart", ... });
  yield { type: "text", text: res.text };
  yield { type: "usage", provider: res.usage.provider, model: res.usage.model, costUsd: res.usage.costUsd };
};

const chat = createChat({ model, tools, can, spend: { limitUsd: 0.50 } });

Silence refuses. This is the whole point.

costUsd is optional, and every ModelFn in the fleet reported nothing when this was written. A cap that reads no number as within budget is a ceiling that can never be reached and never says why — the consumer sets a limit, sees no error, and concludes they are protected.

So the guard has three answers, not two:

| | | |---|---| | ok | counted, under the ceiling | | spend_cap | the ceiling was reached | | unmeasurable_cost | nothing usable arrived — refused | | untrusted_provider | the provider's price is not one we enforce on — refused |

openai and deepseek are absent from TRUSTED_COST_PROVIDERS on purpose. ai-sdk's F040 audit found both cache automatically while the SDK's reported price was too high; gemini and vertex were corrected in 0.32.0 against measured live figures, but openai and deepseek could not be measured — there was no key for either, and ai-sdk wrote nothing rather than writing it from memory. A cap built on a number we invented would be worse than no cap, because it would be believed.

The ceiling never stops the first answer. You cannot know what a call costs before making it, so it bounds the runaway tool→model→tool loop — which is the actual threat — and a single question is always answered. Reaching it arrives as its own limit frame, never as a provider error and never as a stream that simply stops.

The public wall

import { SlidingWindowRateLimiter } from "@broberg/apikey";
import { verifyTurnstile, hashIp } from "@broberg/forms-turnstile/server";

export const POST = toNextHandler(createChatHandler({
  chat,                                  // must carry `spend` — enforced
  mode: "public",
  getCaller: () => ({ anonymous: true }), // NOT null; null is still 401
  guard: {
    rateLimit: {
      limiter: new SlidingWindowRateLimiter({ windowMs: 60_000, max: 10 }),
      keyFor: (req) => hashIp(req.headers.get("x-forwarded-for") ?? ""),
    },
    turnstile: { verify: (t) => verifyTurnstile(t, process.env.TURNSTILE_SECRET!) },
  },
}));

Nothing here is re-rolled — the window is @broberg/apikey's and the bot wall is @broberg/forms-turnstile's, both taken structurally so you inherit no version pin from us. Both are installed here as devDependencies purely so a test proves those shapes still match, rather than asserting it in prose.

Three things worth knowing:

• A public deployment cannot be constructed with a hole in it. No rate limit, no Turnstile, or a capped-looking chat with no ceiling — each throws at construction. "Must exist before the first public deploy, not after the first bill" only means something if it is enforceable.
• The rate limit runs first, because it is local and free. A flood is refused without paying Cloudflare a round-trip per bot.
• A Turnstile outage fails CLOSED, with its own 503. Failing open would be exactly backwards: an outage is the cheapest possible moment for a flood, and the thing behind this wall costs money per request. A rejection tells a real person she is not human; an outage is not her fault, and the two must not share a status.

Test it against something that can say no (/testing)

Every test in this package used to run against a permissive stub, and that is how a defect reached production. cms took the chat live on 0.3.0 and for ~40 minutes every question requiring a tool ended in an error at the user:

event: tool_call    site_summary
event: tool_result  (correct: 48 posts, 22 pages, …)
event: error        mistral 400 — "Unexpected role 'tool' after role 'user'"
                    type=invalid_request_message_order code=3230

The tool ran. The answer came back. The round that would have turned it into a sentence was rejected. 181 tests here and 1,283 in their repo were green throughout, and none of them could have caught it — a fake ModelFn accepts any message shape you hand it.

cms named it better than we did: "attrappen var tro mod GRÆNSEFLADEN og ikke mod VERDEN." The broken condition belongs to the provider, and nothing in either repo had ever spoken to one.

So /testing ships the providers' ordering rule as an assertion:

import { createStrictModel, assertProviderTranscript } from "@broberg/chat/testing";

const model = createStrictModel([
  [{ type: "tool-call", id: "c1", name: "search", args: {} }],  // round 1
  [{ type: "text", text: "48 posts" }],                         // round 2
]);
// …or check a transcript you built yourself:
assertProviderTranscript(messages);   // throws InvalidTranscriptError

Use it in any test that involves tools. It refuses a tool message that answers no assistant turn, a toolCallId matching no call, and a turn taken while calls are unanswered — the same things Mistral, OpenAI and Anthropic refuse.

Measured on our own history, running the pre-fix loop:

| suite | against the code that broke production | |---|---| | 181 stub-based tests | all green | | 14 strict tests | 7 red |

⚠️ It is not a measurement. We have no provider key outside production, so this encodes what the providers document, not what they were observed to do. A live tool round-trip against the cheapest model is still the test that would have caught this first, and it remains outstanding.

⚠️ What it does NOT cover — read this before concluding "tools are tested"

cms hit a second crash minutes after adopting the fix above, and assertProviderTranscript was green throughout:

invalid_type · expected object · received undefined
path: messages.1.toolCalls.0.arguments · "Required"

We emit toolCalls: [{ id, name, args }]. The SDK they hand it to wants arguments. This module validates our shape and the providers' ordering rules — it knows nothing about the library you pass the result to.

A strict double is only strict about the contract it was TOLD about, and there is usually more than one. The guard against "your stub agrees with you" is itself a stub for everything nobody described to it.

It covers exactly one seam: your transcript ↔ the providers' rules. The seam your code ↔ your own SDK is yours, and nothing here can see it.

And the obvious way to use this wrong, which looks entirely right — cms, reported against themselves: they handed createStrictModel straight to their chat factory, so their own translation layer never ran. Mutating that layer turned nothing red. The test proved our engine emits a valid stream and nothing about their delivery — which is where both crashes were.

Point it at what YOU send:

// give the translation a name, so a mutation of it can go red
const payload = toProviderMessages(messages);
expect(payload[1].toolCalls[0].arguments).toEqual({ q: "x" });   // YOUR contract
assertProviderTranscript(messages);                             // ours + the provider's

This example was wrong when first published, and cms caught it within the hour: it said tool_calls (snake), which is what the providers put on the wire and is the natural thing to write with an API doc open. @broberg/ai-sdk normalises to toolCalls at the boundary — it even exports fromProviderToolCall for exactly that. The paragraph teaching people to check field names had the wrong field name in it, which is the whole point restated: verify the shape against the type file, not against your memory of the API.

done is a frame — do not send your own

The loop emits { type: "done", reason } itself, always, on every exit path. A consumer that appends its own after the loop ends puts two done events in the stream. Harmless for a client that ignores the second; load-bearing for the next one. (cms shipped exactly this and flagged it.)

⚠️ A tool-result can be undefined, and undefined does not serialise

ChatTool.run returns unknown. A handler that returns nothing — an early return;, a void function, a caught error path — produces undefined, and the frame carries it through unchanged, on purpose: only you know whether an empty answer means "no results", "not applicable" or "this handler has a bug".

The trap is that JSON.stringify(undefined) is undefined, not the string "undefined". So this crashes three lines later, on a value that looked fine:

const text = JSON.stringify(frame.result);
if (text.startsWith("{")) …          // TypeError

cms hit exactly that in production (2026-08-28). One tool answering nothing threw into their route's outer catch, which ended the stream with "Chat error" on top of a half-written answer that was fine. One silent tool took the whole turn down.

⚠️ The empty string is not the neutral choice — it is the dangerous one

It is what a consumer reaches for because it looks like it asserts nothing. That is exactly why it fails: it gives the model nothing to contradict, so the model fills the gap.

Measured by cms against a real provider (2026-08-28, mistral-small, 5 runs per candidate, one question — "how many documents are on the site?"). What the user was told when the tool returned:

| the tool returned | what the user was told | |---|---| | "" | invented a number in 3 of 5 — «Der er pt. 12 dokumenter på sitet.», stated as fact | | "(no result)" | 0 of 5 — "I cannot see the number." | | "undefined" | 0 of 5 — the same honest answer |

Five runs demonstrates the failure mode; it does not quantify it. The direction is the point — and it is the opposite of what both of us assumed. "undefined" is ugly and fails safe; "" is tidy and fails silently, in the direction where a customer is told a number nobody computed.

JSON.stringify(frame.result) ?? "(no result)"  // says nothing was found
JSON.stringify(frame.result) ?? ""             // ⚠️ invites invention

The full rule — cms's, after this measurement made them correct their own fix from an hour earlier:

| | | |---|---| | 0 and false | are answers — never merge them with "no answer" | | undefined, null, "" | are "no answer" — and must say so, not merely be empty |

The two halves pull opposite ways, which is why || is wrong and ?? "" is wrong. Use ?? — || would merge "zero documents found" with "no answer at all". And watch null specifically: JSON.stringify(null) is the string "null", which is truthy, so a truthiness check on the serialised result lets it through as an answer.

Our own transcript is unaffected — serialise() already falls back with ?? String(value), so the model never receives a non-string. The hazard is specifically for a consumer serialising the frame.

What is NOT in here

The widget, spend caps, retention, redaction, RAG-over-history. The core is the contract; /next, /hono, /http and /client are the HTTP half, /trail is the knowledge half, and /history keeps a conversation alive.

Retention deserves its own warning. Both stores measured during this design keep whole conversations with no expiry at all — and one of them holds GDPR article 9 health data. This package stores nothing, which is not the same as solving it: whatever you persist, give it a deletion rule that runs, not a column that records an intention.

License

MIT · part of the @broberg/* shared inventory.