ai-tally
v0.5.0
Published
AI Tally for Node.js: meter OpenAI and Anthropic spend with one init() call
Readme
ai-tally (Node.js)
AI Tally for Node.js. One init() call meters every OpenAI and Anthropic call in the process, with
no other code changes. It is the Node counterpart of the Python SDK's aitally.init(), and it puts
the same spans on the same wire: the gateway cannot tell which SDK sent a span.
npm install ai-tallyimport * as tally from "ai-tally";
await tally.init({ key: process.env.TALLY_KEY });From then on, every chat.completions.create, responses.create, embeddings.create and
messages.create call records its model and token counts, and so does every Vercel AI SDK
(ai v7+) model call: generateText, streamText, generateObject and each step of a
tool-calling run, with any AI SDK provider. Prompts and completions never leave your process.
Cost is computed at the gateway from the price catalog.
Next.js with the Vercel AI SDK
// instrumentation.ts
export async function register() {
if (process.env.NEXT_RUNTIME === "nodejs") {
const tally = await import("ai-tally");
await tally.init({ key: process.env.TALLY_KEY });
}
}// app/api/chat/route.ts
import { after } from "next/server";
import * as tally from "ai-tally";
export async function POST(req: Request) {
const user = await getUser(req); // e.g. Supabase auth
after(() => tally.flush()); // deliver after the response is sent
return tally.withAccount(user.orgId, () =>
streamText({
model: anthropic("claude-sonnet-4-5"),
messages,
experimental_telemetry: { functionId: "chat" }, // becomes the feature tag
}).toUIMessageStreamResponse(),
);
}The AI SDK path uses the SDK's own telemetry hook (AI_SDK_TELEMETRY_INTEGRATIONS), a process
global, so it works in a bundled app with no instrument option. Two per-call settings bypass
it: experimental_telemetry: { isEnabled: false } opts the call out of every integration, and
passing telemetry: { integrations: [...] } on a call replaces the global list for that call, so
add Tally's to it or the call is not metered.
Call withAccount inside the route handler: Next.js runs its proxy (middleware) in a separate
context, so an account set there does not reach the handler. Pass null when a request has no
signed-in user: the call is still recorded, with no customer.
Server actions, background work and shared helpers
A server action that calls a model or fal needs the same two lines as a route: withAccount and
after(() => tally.flush()).
Work that runs after the response, inside after(), should set its own account and schedule its
own flush, inside the function that does the work: whether the route's account carries into an
after() callback depends on how Next.js runs it. after() can be nested, so scheduling the flush
with after() is safe whether the function runs during a request or after one.
export async function generateLookLayFlat(userId: string, input: LookInput) {
return tally.withAccount(userId, () =>
tally.withTrace({ featureTag: "closet-lay-flat" }, async () => {
after(() => tally.flush()); // never `await tally.flush()` here: see Serverless
const prompt = await generateText({ /* ... */ });
return fal.subscribe("fal-ai/flux/dev", { input: { prompt: prompt.text } });
}),
);
}When one helper makes both a model call and a fal call, or has several callers, wrap its body once
in withTrace. A withTrace tag overrides functionId, so both calls land under one feature.
Give two flows separate feature names when they run on different models (Sonnet for planning,
Haiku elsewhere), so their cost per call is not averaged together.
Steps and tool calls per turn
Each step of a multi-step generateText or streamText run is its own model call, and each is
recorded as its own span. Wrap a turn in withTrace (or withAccount) and all of its spans share
one trace id, so steps per turn is the number of model-call spans in that trace.
Outside such a scope every AI SDK span gets a trace id of its own, marked
tally.aisdk.trace_minted. Per-turn reads leave those calls out rather than count each as a
one-step turn. The gateway's head meter does not read that key, so an untraced AI SDK model call
still counts as one trace in the usage display and the free-tier trace count, as it always has.
Every tool span (below), traced or not, carries gen_ai.trace_id_synthetic, the marker the head
meter skips, so turning tool calls on adds no traces there: a traced turn is already
counted by its model calls. Patched OpenAI and Anthropic clients and fal set neither marker.
Tool calls are recorded too when you turn them on:
await tally.init({
key: process.env.TALLY_KEY,
aiSdkTools: true,
aiSdkToolPrices: { search: 0, lookupOrder: 0, webSearch: 5_000 }, // micro-USD per call
});
await tally.withTrace({ featureTag: "research-agent" }, () =>
generateText({ model, tools, stopWhen: stepCountIs(10), prompt }),
);Each tool execution becomes a tool span with the tool's name and whether it threw
(tally.aisdk.tool_error). The tool's arguments, its result, its error and the messages that led
to it are never read. The outcome is not the OTel status, so a tool error the model recovered from
does not mark the turn a failed run. Every AI SDK model-call span also says whether tool calls were
being recorded (tally.aisdk.tools_recorded), so a turn recorded with the option off reads as
"tool calls unknown", not as a turn with none.
What a tool span costs. A tool's spend on a model or fal call is already on that call's own span,
and an outside API it calls is not known here, so a tool is priced only by aiSdkToolPrices: give
a tool that is pure local code 0, and a paid API its price per call. A tool you do not list is
unpriced (a blank, never $0), and every total that includes it is marked as a lower bound or
withheld. That reaches the cost per customer and per user, the layer split, account margins and a
feature's cost over the window. So list every tool you turn this on for. A tenant price override
for provider ai-sdk and the tool's name also prices it.
What is counted, and what is not:
- Only tools that run in your process: the AI SDK reports a tool through
onToolExecutionEndafter itsexecuteresolves or throws. Provider-executed tools and tools with noexecuteare not seen. - A sub-agent (a
generateTextinside a tool'sexecute) counts toward the parent turn only when it runs in the parent's trace, that is, when it is not wrapped in awithTraceof its own. Then its model calls are steps of that turn and its own tool calls are tool calls of that turn, and the turn keeps the feature of its first model call.withTracealways starts a new trace id, so a sub-agent inside its ownwithTraceis a separate turn, and its steps and tool calls are not counted in the parent's.withAccountkeeps the trace it runs in. - A model call routed through the Tally proxy is metered by the proxy, under the proxy's own trace id, so it is not a step of the turn here. Its run's tool spans are still sent, with no feature tag: the SDK sends no model-call span for that run, so a tagged tool span could add a feature to the head meter that nothing else did. A priced tool in such a run is costed under no feature.
- The agent run view counts every span in a trace, so it shows a turn's tool spans as steps too.
Cost per customer and per feature
Wrap each request once, usually in middleware:
app.use((req, res, next) => tally.withAccount(req.user.orgId, next, { label: req.user.orgName }));
await tally.withTrace({ featureTag: "support-chat", sessionId }, () => answer(question));The account id is HMAC-SHA256 hashed under your tenant's key before it is sent, exactly as the
Python SDK hashes it, so the same customer lines up across both. label is optional and is the
only readable name that travels. tally.unmetered(fn) skips calls you do not want counted at all,
such as a health check. For evals, tag them instead (below), so their cost stays visible.
fal.ai image generation
init() also patches @fal-ai/client, so existing fal.subscribe(...), fal.run(...) and
fal.queue.* calls are metered with no change at the call site. fal bills per image or per output
megapixel, so each call lands as a tool span with its cost:
| Endpoint | Rate (read from fal.ai) |
|---|---|
| fal-ai/flux/dev, fal-ai/flux-1/dev, fal-ai/flux/krea | $0.025 per megapixel, each image rounded up |
| fal-ai/flux/schnell, fal-ai/flux-1/schnell | $0.003 per megapixel, each image rounded up |
| fal-ai/flux-lora | $0.035 per megapixel, each image rounded up |
| fal-ai/flux-pro/v1.1 | $0.04 per megapixel, each image rounded up |
| fal-ai/flux-pro/v1.1-ultra | $0.06 per image |
| fal-ai/flux-2-pro | $0.03 first megapixel, $0.015 each extra, per image |
| fal-ai/nano-banana-2 | $0.08 per image at 1K; 0.75x at 0.5K, 1.5x at 2K, 2x at 4K; +$0.002 for high thinking |
fal-ai/flux/dev, fal-ai/flux-2-pro and fal-ai/nano-banana-2 were read on 2026-11-07, the
rest on 2026-11-10. A megapixel is 1024 x 1024, as fal defines it, and each image is rounded up
to a whole megapixel, so every image bills at least 1 MP: a 512 x 512 render costs the same as a
1024 x 1024 one. Any other endpoint, including the /edit variants (which also bill input
megapixels) and FLUX.2 [dev] (fal-ai/flux-2), is recorded with no cost until you price it:
await tally.init({ key, falPrices: { "fal-ai/flux-2-pro/edit": 45_000 } }); // micro-USD per imageWeb search grounding on nano-banana-2 (+$0.015) is not visible in the request, so a call that
uses it is under-priced by that amount. fal notes its prices can change; pass falPrices to
override any default.
Failed calls
Opt in with init({ falErrors: true }) and a fal call that throws is recorded too, as a tool
span with error status, the endpoint, the HTTP status and fal's request id when known. The error
is rethrown unchanged either way. It is off by default because some views still read a retried
call that later worked as a failed run (below). What it costs
follows fal's own rules (FAQ,
reliability):
| Failure | Cost | Why |
|---|---|---|
| HTTP 5xx | 0, tagged tally.billed=false | fal never charges server errors |
| HTTP 4xx, such as a 422 | none, tally.cost_unknown_reason=may_be_billed | fal may charge if a runner spent GPU time |
| timeout or network error, no status | none, tally.cost_unknown_reason=billing_unknown | nothing says whether a runner ran |
A call your code retries shows up as one error span per failed attempt plus one priced span for
the attempt that worked, so the retries are visible and only the render carries a cost. The fal
client also retries some failures itself inside one call (429, 502, 503, 504 and network errors,
plus 500 on the status polls of subscribe); those attempts are invisible to the SDK, so a call
that recovers there leaves no error span and one that does not leaves one. A request is recorded
failed at most once, and a request fetched twice with fal.queue.result is still metered once.
An error span is marked gen_ai.trace_id_synthetic and carries no feature tag, so failed calls add
no trace and no feature to your usage counts: a trace and its feature count through its successful
calls, as they did before failed calls were recorded. A traced failure stays tied to its run by its
trace id.
tally.billed is stored in the span's attributes as the string 'False' (the gateway's
spelling of a bool), so query it as 'False', not 'false'. Views that judge a whole trace by
its worst status, such as paid-for-nothing waste, still read a trace that retried a fal call and
then succeeded as a failed run.
Evals
Tag eval runs so they show up as their own feature instead of being mixed into product spend:
await tally.init({ key: process.env.TALLY_KEY });
await tally.withTrace({ featureTag: "evals" }, () => runEvalSuite());
await tally.shutdown(); // a script exits right away; this sends the last spans firstWith no withAccount, eval spend is attached to no customer, so it never inflates what a
customer appears to cost.
Under Vitest, start and stop AI Tally in a setupFiles file, in the same worker process that runs
the evals. Vitest runs globalSetup and globalTeardown in the main process, so a shutdown()
there sends nothing. setupFiles run once per test file, and a repeat init() sends the previous
batch first, so nothing is lost.
// scripts/eval/setup.ts, listed in setupFiles
import { afterAll } from "vitest";
import * as tally from "ai-tally";
await tally.init({ key: process.env.TALLY_KEY });
afterAll(() => tally.shutdown()); // stops within 10 s, inside Vitest's 30 s hook limitImporting history
tally-import loads an export of the app's own usage log (CSV or JSON lines), so the dashboard
starts with the history you already have. Nothing changes in the app.
npx tally-import ai_usage_log.csv --until 2026-11-10T00:00:00Z # dry run: prints totals
TALLY_KEY=... npx tally-import ai_usage_log.csv --until 2026-11-10T00:00:00Z --send--untilis required: set it to wheninit()went live, so no call is counted twice.- The dry run prints calls and tokens per day per model. Check them against your own table before sending.
- Columns are guessed from common names (
created_at,model,input_tokens,org_id,feature, ...). Override with--map timestamp=logged_at,account=workspace_id. --input-tokens total(the default) means the input column already includes cache tokens, as the AI SDK'susage.inputTokensdoes. Useuncachedif it holds Anthropic's rawinput_tokens, which excludes them.- Re-running is safe through a ledger: every span the gateway accepts is recorded in
<file>.tally-import.json(or--ledger <path>), and a later run skips those rows. Keep the ledger with the export. Without it a second--senddoubles the dashboard's totals, because its rollups count every insert even after the raw span table collapses the copy. - Timestamps with no zone (the usual Postgres export,
2026-11-09 23:30:00) are read as UTC, like--until.+00and+0530style zones are accepted. - A row whose input or output count is missing or not a plain number is skipped and counted
(
badTokens), never sent as zero. The dry run warns when input counts look like Anthropic's rawinput_tokens(smaller than the row's own cache reads). - Each row is priced at the catalog rate for its own date, so Tally's figure can differ from the price your table stored at the time. History rows carry no cache lifetime, so their cache writes price at the input rate; live spans carry the 5-minute and 1-hour split and price each at its own tier.
Bundled apps (Next.js, esbuild, webpack)
A bundler inlines its own copy of openai or @fal-ai/client, which init() cannot find in
node_modules. Pass what you import (the AI SDK path needs nothing; it is a process global):
import OpenAI from "openai";
import Anthropic from "@anthropic-ai/sdk";
import { fal } from "@fal-ai/client";
await tally.init({ key: process.env.TALLY_KEY, instrument: { OpenAI, Anthropic, fal } });In Next.js, the instrumentation file and each route can end up with separate bundled copies, so
the dependable fix is one line in next.config.ts that keeps the package in node_modules,
where init() patches it automatically:
const nextConfig = { serverExternalPackages: ["@fal-ai/client"] };Check tally.stats().patched after init() to confirm what was patched.
In Next.js, put this in instrumentation.ts under register().
Serverless
Vercel, Lambda and Cloud Run can freeze the process right after a response, before the background flush runs. Hand the flush to the platform, after the response:
import { after } from "next/server";
after(() => tally.flush()); // Next.js route handlers, server actions, nested after()
import { waitUntil } from "@vercel/functions";
waitUntil(tally.flush()); // Vercel outside Next.jsafter() throws outside a request, as in a unit test that calls a route handler directly. If
your tests do that, wrap it:
export function scheduleTallyFlush(): void {
try {
after(() => tally.flush());
} catch {
void tally.flush(); // no request scope (tests, scripts): send in the background instead
}
}Don't await tally.flush() in a function that also serves a request. A slow gateway would add
the flush's time to that request, and a request that times out after paid work is done (an image
already generated, say) invites a retry that pays twice. after() takes the wait off the request.
Every flush has a time limit: 5 seconds by default, set with flushTimeoutMs in init() or
tally.flush({ timeoutMs }) per call. That bounds how long Vercel keeps a function running, and
billed, after the response. A flush that runs out of time keeps its batch and the next flush
resends it under the same batch id, so the gateway counts it once even if the cut-off request did
arrive. tally.shutdown() stops within 10 seconds (shutdown({ timeoutMs })), and anything still
unsent then is counted in stats().undelivered.
On Lambda, await tally.flush() at the end of the handler: the response is already decided there,
and the time limit keeps the handler short.
Streaming
Streams are passed through untouched. Usage is read from the chunks as your code consumes them, and the span is sent once the stream ends, errors or is abandoned.
OpenAI only reports usage on a streamed Chat Completions call when the request sets
stream_options: { include_usage: true }. Set it yourself, or pass streamUsage: true to
init() to have it added. Without it, the span records the model with blank token counts. It
never records a zero. Anthropic streams and the Responses API always report usage.
Replay samples on your own server
To compare other models on your real traffic without your prompts leaving your servers, the SDK
can save a small sample of calls on your own machine. You don't need to change your code for it:
an admin sets a folder in AI Tally and turns on local capture for your organization, and every
app that calls tally.init() starts saving samples there. It saves only while local capture is
on and a folder is set, and logs nothing otherwise.
The folder is the first of these that is set:
replayDirin code:await tally.init({ key: process.env.TALLY_KEY, replayDir: "/var/lib/tally/replay" }).- The
TALLY_REPLAY_DIRenvironment variable. - The folder an admin set in AI Tally. The SDK checks it again before it writes anything there.
The folder must belong to the user your app runs as, and no one else may write to it. The
folder that holds it must belong to that user or to root, and no one else may write to it
either, unless everyone may and it has the sticky bit. Its path may use only ASCII letters,
digits,
.,_and-, and must not be a symlink or a system folder such as/etc,/usror/tmp. On macOS a folder under/varor/tmpresolves into/privateand is refused too. An existing folder must be empty or hold only the SDK's samples. A folder that doesn't exist yet is created only if the folder that holds it exists, and only the user your app runs as can read it. Anything else is refused with a warning. A folder, or the folder that holds it, that its group may write to is refused too, so on a system with umask 002 or user private groups, create the folder with mode 0700 first, or pick a parent with mode 0755. On Windows a folder set in AI Tally isn't used; setreplayDirorTALLY_REPLAY_DIRthere.
On a serverless runtime (Vercel, AWS Lambda, Netlify, Azure Functions, Google Cloud Run and Cloud Functions) local capture doesn't start on its own.
The SDK reads these settings with Node's own http agents, so a proxy set with NODE_USE_ENV_PROXY
applies to it; a custom undici dispatcher doesn't, unless you pass fetch to tally.init().
By default it saves 1 in 100 eligible calls (replaySampleRate) and keeps them for 7 days
(replayRetentionDays). Eligible calls are OpenAI and Anthropic calls through their own client
libraries that aren't streamed, and Vercel AI SDK generateText and streamText calls to an
OpenAI or Anthropic model, streamed or not. Nothing is sent to AI Tally. The files hold real
prompts and answers, so treat that folder like any other customer data. Provider keys and
request headers are never written.
An AI SDK run that calls tools is saved once, at the step that answers in text, with the whole
conversation up to it: the tool calls, what your tools returned, and the tools' names,
descriptions and input schemas. Images and files are written as a placeholder, never their
content. A call that stopped early (cut off, filtered, failed, or ended in a tool call) isn't
saved, nor one with recordInputs or recordOutputs set to false, nor one that offers a tool
the provider runs itself. AI SDK calls are only saved locally, never in hosted mode.
fal calls are saved too, at most 2 per server per UTC day, one in each 12-hour UTC window
(falSamplesPerDay sets how many, splitting the day into that many windows; 0 saves none):
a sampled fal.run, or fal.queue.submit then fal.queue.result (as fal.subscribe does), to
an endpoint AI Tally prices. The sample holds the input your app passed and the output images,
which the SDK downloads after the call returns, only over https from fal's media hosts and at
most 10 MB each, and saves in the day's images folder with mode 0600. A call whose input holds a
file or a data: URI isn't saved, nor one whose download fails. Retention removes the images with
their day, and when local capture follows the admin settings alone (no replayCapture in code)
in a folder set in AI Tally, only the ones this machine saved. The limit is shared by every
process on the server: each window is a small slot file in that day's images folder, which one
process claims. tally eval --models fal-ai/flux/schnell,... (Python SDK) reruns these samples on
other fal endpoints and has a judge compare the images.
The folder layout is the Python SDK's, so tally eval --samples /var/lib/tally/replay (from the
ai-tally Python package) compares models on it, and tally eval purge deletes it. The SDK
itself deletes only its own sample files, by retention, and in a folder set in AI Tally only the
ones this machine wrote.
replayCapture: false turns local capture off whatever the admin settings say.
replayCapture: true (or "local") asks for it in code, and also works on serverless. Samples
are sent to AI Tally only if you write replayCapture: "hosted". Any other value turns capture
off with a warning.
Options
| Option | Default | |
|---|---|---|
| key | TALLY_KEY | Ingest key. Without one, init() warns and meters nothing. |
| endpoint | TALLY_ENDPOINT, then https://ingest.ai-tally.com | |
| instrument | none | Provider classes to patch explicitly. |
| autoPatch | true | Find openai and @anthropic-ai/sdk in node_modules, both the CommonJS and ES module copies. |
| aiSdk | true | Meter Vercel AI SDK calls through its telemetry hook. |
| aiSdkTools | false | Also record each AI SDK tool execution as a tool span: name and whether it threw, only. |
| aiSdkToolPrices | {} | Micro-USD per call by tool name for those spans; 0 for local code. Unlisted tools are unpriced. |
| falPrices | built-in rates above | Per-endpoint fal prices: micro-USD per image, or a function of the images and input. |
| falErrors | false | Record failed fal calls as error spans (see Failed calls). |
| streamUsage | false | Add include_usage to streamed Chat Completions calls. |
| hmacWaitMs | 2000 | How long a span waits for the tenant's HMAC key before going out without an account. A flush() or shutdown() running meanwhile extends the wait to just before its own time limit. A key the gateway refuses (401 or 403) is not waited for. |
| flushTimeoutMs | 5000 | Longest one flush may take, the 2-second background flush included. Unsent spans stay queued. |
| replayCapture | follows the admin settings | Not set: save sampled calls on this machine while an admin has turned on local capture (never on its own on serverless). false: never. true or "local": save them, serverless included. "hosted": send them to AI Tally. |
| replayDir | TALLY_REPLAY_DIR, then the folder set in AI Tally (not on Windows) | Where local samples are saved. |
| replaySampleRate | 0.01 | Share of eligible calls saved locally. |
| replayRetentionDays | 7 | Days local samples are kept. |
| falSamplesPerDay | 2 | fal calls saved locally per server per UTC day, one per equal window of the day, with their images. 0 saves none. |
| allowInsecureHttp | false | Allow a non-TLS endpoint, for local dev only. |
| onError | one warning per error site | Every internal error lands here. Nothing is ever thrown into your code. |
What it does not do
- Calls read with
.asResponse(). The SDK observes the parsed result, so a caller that only reads the rawResponseis not metered. Use.withResponse()to keep both. - Calls through the AI Tally proxy are skipped on purpose (a Tally proxy
baseURL, or anX-Tenant-Keyheader), because the proxy already metered them. For the Vercel AI SDK this is seen on the outgoing request, through a hook onglobalThis.fetchthat reads only the URL and that header. A provider given its ownfetchoption bypasses the hook, so a proxied call made that way is counted twice; so is a proxiedgenerateObject, which has no per-call wrapper. - Other libraries. Raw
fetch, the official Gemini and Bedrock clients, and LangChain's own clients are not patched yet. Use the proxy or the OpenTelemetry path for those. - Other image providers. fal.ai is covered (below); AI SDK embeddings and image models, and
fal's
fal.stream, are not metered yet. - Streams nobody reads. The span is sent when the stream is consumed or closed. A stream that is never read is never recorded.
How it works
init() replaces create on each resource's prototype, so every client is covered, including
clients built before init(). Every helper (parse, stream, messages.stream) goes through
create, so each call is metered exactly once. The wrapper hooks the response's parse step rather
than attaching its own .then(), which would force a parse and race a caller for the response
body, and it follows the promises the helpers derive from it (_thenUnwrap), since openai 7
parses those without going through the original's parse step.
Spans are batched in the background and retried with byte-identical bodies, so the gateway dedupes a retry instead of double-counting it. A 200 that sheds some spans is read item by item, as the Python SDK does: retryable ones are resent, refused ones are counted, and each request times out after 5 seconds. A flush stops at its time limit, capping each request's timeout and each wait at the time left; the batch it was on keeps its bytes, its attempt count and the wait the gateway asked for, so the next flush picks it up exactly where this one stopped. The account-hashing key is retried every minute until it loads and refreshed every 12 minutes.
Development
npm ci
npm run typecheck
npm test # includes the cross-language contract test when uv is installed
npm run buildtest/contract.test.ts runs the spans this SDK emits through the Python schema validator, the
gateway's SpanValidator, cost enrichment and row mapping, and checks the account hash against
the Python implementation.
