@dapplelight/signet-ai
v0.5.0
Published
Provider-agnostic structured LLM calls with budget caps, caching and failover.
Maintainers
Readme
@dapplelight/signet-ai
Provider-agnostic structured LLM calls with budget caps, caching and failover. Resolves signet#1.
The extractor is structured output only. Every extract call goes through
a Zod schema and returns a typed, validated object, never free text. Tool loops,
streaming chat, embeddings and grounded answers live on their own subpaths (see
"AI SDK helpers" below), so the core stays small and needs no ai install.
import { createExtractor, memoryCache, createBudgetLedger } from "@dapplelight/signet-ai"
// The AI SDK adapter is a SUBPATH, not on the barrel — see "Why the adapter is
// a separate import" below. Requires `ai` 7 or later.
import { aiSdkProvider } from "@dapplelight/signet-ai/providers/ai-sdk"
import { createWorkersAI } from "workers-ai-provider"
const workersAi = createWorkersAI({ binding: env.AI })
const ai = createExtractor({
providers: [
aiSdkProvider({ name: "workers-ai", model: workersAi("@cf/meta/llama-3.3-70b-instruct-fp8-fast"),
pricing: { perCallUsd: 0.0005 } }), // an estimate: Workers AI bills in Neurons
aiSdkProvider({ name: "backup", model: someOtherModel,
pricing: { inputPerMTokens: 0.15, outputPerMTokens: 0.6 } }),
],
cache: memoryCache(),
// $5/day, fails closed. Memory is per isolate: see "Budgets across isolates".
ledger: createBudgetLedger(5, 86_400),
})
const { value, meta } = await ai.extract({
schema: intentSchema,
input: "8000 se kam ka snowboard",
instructions: "Pull the budget, size and category from this shopping request.",
budget: { maxCostUsd: 0.002 },
})
// meta: { provider, model, usage, costUsd, cacheHit, attempts, durationMs }Model choice matters more than anything in this package
Benchmarked through this extractor on real inputs by the first production consumer, five free Workers AI models, same work:
| Model | Latency |
|---|---|
| @cf/meta/llama-3.3-70b-instruct-fp8-fast | 1.1s |
| @cf/zai-org/glm-4.7-flash | 17.7s / 30.0s / 21.0s, one ALL_PROVIDERS_FAILED |
| @cf/meta/llama-3.1-8b-instruct | fails instantly — no structured output on this path |
glm-4.7-flash was this README's example and is unusable inline. If you copied
it before, change it. End-to-end latency in production went from 17–30s to
1.6–3.0s including a Shopify call.
A model with no structured-output support fails closed in ~0.1s, which is the correct behaviour — but a provider array containing only that model will always fail.
Schema validity is not truth
This package guarantees shape, not correctness. "Typed and validated or it is an error" is easy to read as "correct". It is not the same claim, and the difference matters.
Observed in production, all schema-valid and all wrong:
maxPrice: 8000for "snowboards for a beginner" — no price in the input- a field meant to hold
"ski jacket"returned "The user asked for a warm skiing item with a maximum price of $900…" - every model returned
inStock: truefor requests that never mentioned availability
Re-validation passes all of these, correctly — they are the right shape with invented content. If a hallucinated value would cause harm, verify each claim against the input text and discard what the text does not support. That layer is application-specific and deliberately not in this package.
Decisions that are load-bearing
Providers are an array, tried in order. Adding or reordering is config, not code. Exhausting Workers AI's free allocation (error 3036, delivered as a 429) fails over to the next entry with no product change.
Budget fails closed. Once a window is exhausted, calls are refused. A
fallback value does not mask a budget refusal — running out of money is a
configuration problem the caller must see, not a soft failure.
A ledger needs prices. With a ledger set, createExtractor throws
AI_UNPRICED_PROVIDER if any provider has no pricing, because an unpriced
provider counts as $0 and the cap never fills. Use perCallUsd for flat-rate
providers, or pass allowUnpriced: true when one really is free.
Only transient errors retry. A schema violation or a bad key fails identically every time; retrying spends budget and latency for a guaranteed failure. Unknown error shapes are treated as permanent — "retry on doubt" is how a cap gets spent on something that was never going to work.
The SDK's own retries are disabled (maxRetries: 0). Retrying in both
places multiplies spend silently underneath a cap.
Output is re-validated against the schema even though the provider claims to have done it. A provider returning an unvalidated object is the exact failure this package exists to prevent.
The schema and the model are part of the cache key. The same prompt against
a changed schema is a different question; reusing the old answer returns data
that no longer validates. Switching models should not keep serving the old
one's answers either, so the key also covers the provider chain (name/model
of each provider, in order).
Persistent caches
memoryCache() lives and dies with one isolate. For answers that survive a
restart and are shared everywhere, pass a persistent Cache; it is the same
two-method interface (get, set(key, entry, ttlSeconds)), so your own store
works too.
// Workers KV: the namespace binding as is (or any KeyValueLike, e.g. upstashKv)
createExtractor({ providers, cache: kvCache(env.AI_CACHE), cacheTtlSeconds: 86_400 })
// A table in your database (SQLite, D1 or Postgres), from @dapplelight/signet-db
import { dbCache } from "@dapplelight/signet-db"
createExtractor({ providers, cache: dbCache(db, schema.aiCache) })Entries carry their own expiry, checked on read, so a store that keeps a key
longer (Workers KV's 60-second minimum) still expires it on time. A cache that
is down reads as a miss and a failed write is dropped (both emit
ai.cache_error to onEvent): an answer already paid for is never thrown away.
Each hit is reported to onUsage with cacheHit: true, costUsd: 0 and
savedUsd, what the original call cost.
Budgets across isolates
The default ledger keeps spend in memory, in one isolate. On Workers each isolate has its own and a cold start empties it, so "$5/day" really means $5 per isolate per cold start. Give the ledger a shared store:
// Exact: one Durable Object per budget key. Free-plan Workers include Durable Objects.
import { BudgetCounter, durableObjectBudgetStore } from "@dapplelight/signet-ai/budget-do"
export { BudgetCounter } // from the worker entry, so the binding finds the class
const ledger = createBudgetLedger(5, 86_400, {
store: durableObjectBudgetStore(env.AI_BUDGET),
perKeyUsd: 0.5, // each user gets $0.50/day within the $5
})
await ai.extract({ ...args, budget: { key: userId } })// wrangler.jsonc
"durable_objects": { "bindings": [{ "name": "AI_BUDGET", "class_name": "BudgetCounter" }] },
"migrations": [{ "tag": "v1", "new_sqlite_classes": ["BudgetCounter"] }]kvBudgetStore(env.KV) is the cheap alternative, and it is approximate: its
window is fixed (it resets on the boundary) and concurrent writes can lose a
record. Any object with spent and record is a BudgetStore, for Redis or a
database off Workers.
Calls in flight are not reserved, so concurrent calls can overshoot a cap by their own cost.
Usage reporting
Every call can report its tokens and cost. Pass onUsage to createExtractor
or to any helper below; each call that reached a model reports once:
import { analyticsUsage } from "@dapplelight/signet-ai"
const ai = createExtractor({
providers,
onUsage: (r) => log.info("ai.usage", r),
// or straight to an AnalyticsSink from @dapplelight/signet-server/analytics:
// onUsage: analyticsUsage(sink),
})A report is { operation, provider, model, usage, costUsd, key, durationMs }.
operation is extract, tools, stream, embed or ground; key is the
budget key (a user id), so cost can be attributed per user. A cache hit is
reported with cacheHit: true, costUsd: 0 and savedUsd (the original
call's cost), so the cache's savings add up beside the spend. Fallbacks spend
nothing and are not reported. A call refused by the per-call cap
after the spend is reported, because the money is gone. A hook that throws, or a
sink that rejects, is ignored: analytics must not fail a paid call.
analyticsUsage(sink, name?) writes one ai.usage event per report, with
cacheHit: 1 and savedUsd on a hit.
AI SDK helpers
Requires ai 7 or later. Each helper takes a model from any AI SDK provider
package, so nothing here names a provider. Each takes onUsage, pricing,
key, provider (the label in reports) and abortSignal, and turns the SDK's
own retries off, as the extractor does.
Tool loops
import { runTools } from "@dapplelight/signet-ai/chat"
import { tool } from "ai"
const r = await runTools({
model,
tools: { lookupOrder: tool({ description, inputSchema, execute }) },
prompt: "Where is order A1?",
maxSteps: 5, // default 5; each model call is a step
onUsage,
})
if (r.capped) { /* stopped at maxSteps while the model still wanted tools */ }
// { text, steps, capped, toolCalls, usage, costUsd }usage is the total over every step. A perCallUsd price is charged once per
step. Check capped: a capped run's text is often empty.
Streaming chat
The server returns streamChat(...); the client reads it with useSignetChat,
which wraps useChat from @ai-sdk/react (an optional peer, install it in the
web app).
// server (Hono, Workers, Next route handler: anything that returns a Response)
import { streamChat } from "@dapplelight/signet-ai/chat"
app.post("/api/chat", async (c) => {
const { messages } = await c.req.json()
return streamChat({ model, messages, tools, maxSteps: 5, onUsage, key: userId,
abortSignal: c.req.raw.signal })
})// client
import { useSignetChat } from "@dapplelight/signet-ai/react"
const { messages, sendMessage, status, stop, usage } = useSignetChat({ api: "/api/chat" })Usage reaches onUsage on the server when the stream ends, and the client as
message.metadata.usage on each assistant message; usage from the hook is the
conversation's sum. api and headers are read once per chat id, as in
useChat; pass headers as a function for a value that changes. Authenticate
the route and cap messages yourself: the client sends the whole history.
messages is untrusted input. streamChat drops system messages (set
instructions with system) and every tool part from the history, so a browser
cannot plant a system prompt or a fake tool result. Tools the model calls during
the request still run; their results from earlier turns are not replayed.
Embeddings and search
import { embedDocuments, searchDocuments } from "@dapplelight/signet-ai/ground"
const indexed = await embedDocuments(embeddingModel, docs) // [{ id, text, embedding }]
// store `indexed` anywhere (JSON column, KV); vectors are plain number arrays
const hits = await searchDocuments(embeddingModel, "how long is delivery", indexed, {
topK: 5, minScore: 0.3,
})
// [{ id, text, score }], best firstThere is no vector store: ranking is cosine similarity in memory, which is fine
for a few thousand documents. Past that, use the platform's index (Vectorize,
pgvector) and pass its results to groundedAnswer.
Grounded answers
import { groundedAnswer } from "@dapplelight/signet-ai/ground"
const r = await groundedAnswer({ model, question, documents: hits })
if (!r.grounded) return "I couldn't find that in our help pages."
// r.answer, r.citations: [{ id, quote }], r.rejectedThe model must cite each claim with a document id and a verbatim quote. Every
citation is checked: an unknown id, or a quote that does not appear in that
document (ignoring case and whitespace), goes to rejected. grounded is true
when at least one citation survives. This checks the citations, not the
reasoning: a model can quote a real line and still over-claim, so show the
quotes beside the answer. Documents are wrapped in <document> tags and the
model is told their text is data, which reduces but does not remove prompt
injection from document content.
Testing the helpers
Use the SDK's mock models from ai/test (MockLanguageModelV4,
MockEmbeddingModelV4); test/sdk.ts in this package shows each helper driven
by them with no network.
Why the adapter is a separate import
aiSdkProvider lives at @dapplelight/signet-ai/providers/ai-sdk and is deliberately
not re-exported from the barrel.
export * resolves eagerly in ESM, so barrelling it would make importing
createExtractor alone pull in ai — turning an optional peer into a hard
requirement. Anyone using only fakeProvider in tests, or writing their own
Provider, would need a dependency they never asked for, and would only find
out at runtime.
The core package is transport-agnostic and depends on nothing but
@dapplelight/signet-lib. Same reasoning as @dapplelight/signet-server/http needing
hono only if you use it.
import { createExtractor } from "@dapplelight/signet-ai" // no `ai` needed
import { aiSdkProvider } from "@dapplelight/signet-ai/providers/ai-sdk" // needs `ai`
import { runTools, streamChat } from "@dapplelight/signet-ai/chat" // needs `ai`
import { groundedAnswer } from "@dapplelight/signet-ai/ground" // needs `ai`
import { useSignetChat } from "@dapplelight/signet-ai/react" // needs `ai`, `@ai-sdk/react`Testing without network or spend
import { createExtractor, fakeProvider } from "@dapplelight/signet-ai"
const ai = createExtractor({ providers: [fakeProvider({ budget: 8000, category: "snowboard" })] })fakeProvider takes a fixed value, a list, or a function of the input, and can
be told to fail N times first to exercise retry and failover paths.
Errors
AiError extends ApiError from @dapplelight/signet-lib/api, so an AI failure is handled
like any other API failure. Codes: AI_BUDGET_EXCEEDED (429),
AI_ALL_PROVIDERS_FAILED (502), AI_SCHEMA_INVALID (502), AI_NO_PROVIDERS
(500).
Jev: @dapplelight/signet-ai/jev
A safe wrapper for Jev, TypeSafe's System One classifier: text in, typed decision out. app-a, app-b and app-d each had their own copy with the same model pin, 0.95 floor and 8 s timeout. One of them had no floor and threw.
import { createJev } from "@dapplelight/signet-ai/jev"
// Build it where env exists (a handler, a CLI main), never at module scope.
const jev = createJev({
apiKey: env.JEV_API_KEY, // missing: every call returns its fallback
// model = "jev-1.13.0", minConfidence = 0.95, timeoutMs = 8000
// allowSensitive = false // the explicit opt-in for private/financial text
onEvent: (e) => log.info("jev", e), // redacted: no key, state or question text
})
const d = await jev.choice({
question: "Which spending category fits this bank transaction?",
options: { food: "Restaurants, delivery", travel: "Flights, cabs", unsure: null },
state: `Money paid out. Narration: "${narration}"`,
fallback: "unsure",
sensitive: true, // required: bank text is financial
})
// { value, confident, source: "jev" | "fallback", confidence?, reason?, answer? }
if (d.confident) apply(d.value)
else review.push({ best: d.answer, confidence: d.confidence })| Method | Jev primitive | value |
|---|---|---|
| choice({ question, options, fallback, ... }) | choice, up to 255 labels, as a list or label -> description | one of the labels |
| score({ question, levels, fallback, ... }) | score, 2-10 ordered levels | a number, can fall between levels |
| noul({ question, fallback, ... }) | noul | a boolean; probability carries the raw yes-probability |
| batch({ state, questions, ... }) | several named questions about one state, in one request | a decision per name |
Every call takes state, sensitive, fallback, and optionally minConfidence
and signal. jev.enabled is false without a key, so an app can skip
preparing work it will not send.
It never throws for the failures that matter. Every call resolves to a
decision the app can act on. The fallback is used, with a reason, when:
| reason | When |
|---|---|
| sensitive_not_allowed | sensitive: true and the instance lacks allowSensitive: true. Checked first; fetch is never called |
| no_key | apiKey missing or blank. Fetch is never called |
| low_confidence | below the floor. answer and confidence keep what Jev said, for a review queue |
| timeout | over timeoutMs, even if the fetch ignores its abort signal |
| http_error | a non-2xx response, such as 401 or 429. The event carries status |
| network_error | fetch rejected |
| invalid_response | not JSON, a missing answer, or a label outside the offered set |
| invalid_request | no questions, an out-of-range floor, score levels outside 2-10, or a fallback not among the options |
| aborted | the caller's signal fired |
Decisions that are load-bearing:
- The 0.95 floor is on by default. On app-a's first real run, 0.85-0.94 let
wrong guesses through and 0.95 let none through. Pass
minConfidenceper call where a wrong answer is cheap, as app-b's loop does. sensitiveis required by the type, and a JS caller that omits it is treated as sensitive. The opt-in belongs to the instance, so the owner decides it once in config, not each caller in code. Deciding what is safe to send is still the app's job: app-a strips digit runs from narrations.- The noul confidence is
max(p, 1 - p). Jev reports none for noul, so a 0.7 "yes" falls back at the default floor. Rank byprobabilityif you need to. - Fetch only, no SDK. The API is a single
POST /v1/systemonewith Bearer auth.@typesafe-ai/sdkretries twice by default with a per-attempt timeout, so its "8 s" could reach about 27 s. It also falls back toTYPESAFE_API_KEYfrom env and logs request bodies at debug level. HeretimeoutMsis the total, with no retries, because the fallback is the retry. - The key goes into the Authorization header and nowhere else. Upstream error messages and response bodies are never copied into a result or event, because a transport can echo headers. A test drives every failure path through a transport that tries to leak the key.
- Not on the barrel. Import it from
@dapplelight/signet-ai/jev. It has no dependencies.
Migrating the existing copies
- app-a (
packages/mcp/src/lib/jev-categorize.ts): dropJEV_MODEL,TIMEOUT_MS,makeJevClassifier, theTypeSafeClientimport and the@typesafe-ai/sdkdependency. BuildcreateJev({ apiKey: env.JEV_API_KEY, allowSensitive: env.JEV_ALLOW_PRIVATE_MODE === "1" }), then calljev.choice({ ..., fallback: "unsure", sensitive: Boolean(crypto) }). That replacesassertPrivateModeAllowedinsidejevCategorize, and atry/catchcounting errors becomesreasonnot beinglow_confidence. The confidence comparison moves into the wrapper;low_confidencerows go toreviewusinganswerandconfidence. KeepsanitizeNarration, theunsurelabel and the no-auto-apply rule for transfers: those are product rules. - app-b (
scripts/jev-client.mjs): delete the file.decide()inloop/jev-steps.mjsbecomesjev.choice({ ..., fallback: safe, minConfidence, sensitive: false }). Behaviour fix:jev-pr-risk.mjs,jev-triage.mjsandjev-stop-gate.mjscallclassifywith no floor, so a 0.3 guess labelled a PR, and any error or missing key threw and killed the script. They now get the 0.95 floor and a fallback:mediumfor PR risk,structuralfor the stop gate so E2E still runs, and no tier for triage.assessCompletenessbecomesjev.noul({ ..., fallback: false }). - app-d (
src/lib/jev.ts): deleteJEV_MODEL,MIN_CONFIDENCE,TIMEOUT_MS,getClient, and thetry/catchreturningnullinjevClassify.jevEnabled()becomesjev.enabled. The null-or-low-confidence check inmatch.tsconfident()and inroute.tsbecomesd.confident.jevScorebecomes ajev.batchofnoulquestions per chunk of 25, readingprobabilitywith a fallback oftrueso an unscored video is kept. All app-d calls send public content, so they usesensitive: false.
Out of scope
Chat history storage, vector databases, and agents beyond a single tool loop. Add when a second product needs them, not before.
Test
bun run test runs 62 extractor assertions (failover, retry classification,
budget fail-closed, cache keying, schema enforcement), 59 Jev assertions (the
confident path, every fallback reason, the privacy gate, batch, and the key
never leaking) and 37 AI SDK helper assertions (the adapter, onUsage, tool
loops and the step cap, streaming, embeddings, search and citation checks), all
with no network.
