margin-meter
v0.4.1
Published
Tiny dependency-free client that auto-instruments OpenAI/Anthropic/Gemini and emits LLM call + outcome economics to a Margin ingest API.
Maintainers
Readme
margin-meter (TypeScript SDK)
You probably do not need this to start. Margin's first step installs nothing: sign in at trymargin.io/console, connect the repository your agents live in (or drop the folder), and press Execute. Each agent runs on the evals beside it and every call is metered. This SDK is the optional route for metering your live production traffic, once you want that as well (
/start#production).
The tiny client a TypeScript/JavaScript project imports to connect to
Margin — the TS twin of the Python margin-meter package. It wraps
your LLM calls and records their outcomes, emitting each one over HTTP to a
Margin ingest API (POST /api/ingest/calls / /api/ingest/outcomes),
authenticated with a per-project ingest key.
- Dependency-free. Uses the global
fetch— nothing to install at runtime. - Off your hot path. Calls are buffered and flushed in batches, so your agent's latency never depends on Margin being up. See Batching — read it before you integrate, because the guarantees and their costs are both stated there.
- Fail-safe. A failed emit resolves to
{ ok: false, … }instead of throwing. PassraiseOnError: truefor strict/CI behaviour. - Provenance-honest.
isSimulatedis carried through untouched; the writtensourceis forced server-side to your key's project — you cannot spoof another project's economics.
Install
From the npm registry:
npm install margin-meterCheck the version you get. This package ships from the registry, and the registry can lag the repo. Confirm with
npm ls margin-meter, and compare it againstsdk/ts/package.jsononmain. An older release silently omits newer capture (e.g.substrate, reasoning tokens), which under-reports your spend rather than failing loudly — the one failure mode this SDK exists to prevent.
Unlike the Python SDK, this package is not git-installable:
npm has no equivalent of pip's #subdirectory=, dist/ is a build artifact
that is not committed, and the entry points resolve to ./dist. Installing
from a clone or a git URL therefore yields a package with no entry point. Use
the registry, or publish a new version from sdk/ts (npm publish runs the
build via prepublishOnly).
Configure
Two environment variables — the deployed API base and your project's key:
export MARGIN_INGEST_URL="https://trymargin.io"
export MARGIN_INGEST_KEY="mgk_…" # issued by the Margin ownerThe Margin owner issues your project a key with the provisioning CLI in the
Margin repo (python3 scripts/issue_ingest_key.py <your-project-slug>). The raw
mgk_… key is shown once — only its hash is stored.
Use
import { MarginMeter } from "margin-meter";
const meter = new MarginMeter(); // reads MARGIN_INGEST_URL + MARGIN_INGEST_KEY
// 1) Wrap the LLM call — latency is timed automatically, cost computed server-side.
const answer = await meter.measure(
{ workflowId: "cart-parse", provider: "google", model: "gemini-2.5-flash" },
async (m) => {
const resp = await callTheModel(/* … */);
m.setTokens({ inputTokens: 1200, outputTokens: 300, cacheReadTokens: 800 });
return resp;
},
);
// 2) Record the outcome it produced (the unit of productivity).
await meter.recordOutcome({
workflowId: "cart-parse",
passed: true,
qualityScore: 0.94,
qualityMethod: "ground_truth",
});Or record a call directly:
const res = await meter.recordCall({
workflowId: "cart-parse",
provider: "google",
model: "gemini-2.5-flash",
inputTokens: 1200,
outputTokens: 300,
});
if (!res.ok) console.warn(`margin ingest failed: ${res.error} (${res.statusCode})`);Say what each call was FOR (taskKey)
workflowId names the pipeline and operation names the step inside it. Neither
says what the work actually WAS — a billing question and a password reset look
identical on both. taskKey is that third axis, in your own vocabulary:
await meter.recordCall({
workflowId: "support-triage",
provider: "openai",
model: "gpt-4o-mini",
inputTokens: 812,
outputTokens: 96,
taskKey: ticket.topic, // "billing" | "password-reset" | whatever you already use
});Why it earns a field. Margin measures a cheaper route's parity on a sample and
records what that sample was made of. If the distribution of arriving work moves —
new ticket topics, a mix that shifts as you grow — parity decays with no model
change at all, and nothing that watches for a route getting worse will fire,
because nothing got worse. With taskKey flowing, Margin compares the measured mix
against the arriving one and says steady or moved. Without it the answer is
unknown, which is what the console shows and is the honest state, not a bug.
It is never priced and never rolled into cost. Send it on more than 60% of a workflow's calls and the comparison becomes answerable; below that Margin declines to rule rather than describing your instrumented slice as if it were your traffic.
Say where each call RAN (environment)
environment splits the work you sell from the work you did to decide what to
sell. A spike, an eval pass and your production traffic are three different kinds
of spend, and only one of them belongs in the number you report upward.
await meter.recordCall({
workflowId: "support-triage",
provider: "openai",
model: "gpt-4o-mini",
inputTokens: 812,
outputTokens: 96,
environment: "prod", // "prod" | "dev" | "eval"
});| value | send it for |
| --- | --- |
| prod | traffic your customers cause |
| dev | a spike, a local run, a branch deploy |
| eval | a graded pass, in CI or from an eval harness |
Why it earns a field. costPerOutcome is total call spend divided by passed
outcomes. With environment unset, your R&D and your operations land in that one
number, so the headline describes neither. Margin already keeps its own face-off
spend out of your number; this is how you get the same split on your side.
It does not default. Leave it off and the row is unknown, which is honest and
costs you nothing, where guessing prod would book your spike as production. It
is never priced and never rolled into cost. It is a label, like team.
Multi-agent pipelines (name each stage)
A crew of agents — a planner→coder hand-off, a LangChain pipeline — runs under one
workflow. Auto-instrumentation binds a single operation at wire-up, so every
agent would meter under the same label and the run reads as one blob. stage()
splits it: wrap the current step and every metered call inside — however many
awaits deep — is attributed to that stage, under a shared run id. It is the twin
of the Python SDK's stage(); because TS has no with, it takes the block as a
callback and returns whatever the callback returns.
import { MarginMeter, stage } from "margin-meter";
const meter = new MarginMeter();
const runId = "run-42";
await stage("classify", { taskClass: "signal-check", sessionId: runId }, () =>
classifier.run(ticket), // its calls meter as operation="classify"
);
await stage("implement", { taskClass: "algorithm", sessionId: runId }, () =>
implementer.run(plan), // ... and these as operation="implement"
);The two stages share sessionId but carry distinct operations, so the run is a
real multi-stage pipeline the Money Map decomposes — and taskClass, declared per
step, is what lets margin_stage_plan recommend a model and effort for each. It
uses AsyncLocalStorage (the direct analogue of Python's contextvars), so it stays
correct across await and concurrent runs; it re-labels and links only — tokens,
cost and the outcome are untouched.
Nest stage()/span() to build the tree, and reach for span() when a block fans
out and you want to record the orchestration shape:
import { span } from "margin-meter";
await span({ pattern: "panel", branchWidth: 3, stoppingRule: "majority of 3" }, () =>
Promise.all(candidates.map((c) => judge(c))), // each judge nests its own node
);One caveat carried over from the Python side: AsyncLocalStorage does not cross a
worker boundary (as a contextvar does not cross a bare thread). A branch you run on
a worker must pass an explicit parentSpanId on the call or in the options, or it
records as a root — which reads as "no chain happened" rather than the missing edge
it is.
Auto-instrument (a few lines for a whole repo)
To instrument an existing agent you didn't write, don't edit every call site.
Point the meter at the client the agent already constructed and every model call
is timed and metered automatically — tokens are read straight off each
provider's response, so there's no setTokens bookkeeping:
import OpenAI from "openai";
import { MarginMeter, instrumentOpenAI } from "margin-meter";
const client = new OpenAI();
const meter = new MarginMeter();
const undo = instrumentOpenAI(client, meter, { workflowId: "aider-fix" });
// ... the agent's normal client.chat.completions.create(...) calls are metered ...
await meter.recordOutcome({ workflowId: "aider-fix", passed: testsGreen });
undo(); // restore the original methodinstrumentAnthropic(client, …) patches messages.create;
instrumentGemini(model, …) patches a GenerativeModel's generateContent
(recovering the model id off the instance). Each returns an undo() and throws
MarginInstrumentError if the client shape has moved.
To flow taskKey (above) through the auto-instrument path, pass a resolver, not
a value — one wrapper meters a whole run, so the unit of work is read at call
time, never baked as a constant (a constant would collapse the run into one class):
const undo = instrumentOpenAI(client, meter, {
workflowId: "support-triage",
resolveTaskKey: () => currentTicket()?.topic, // read per call; undefined ⇒ unknown
});Two lower-level primitives when you want tighter control:
// Wrap ONE call (no monkeypatch) — returns a drop-in replacement.
const create = meter.wrap(client.chat.completions.create.bind(client.chat.completions), {
workflowId: "cart-parse",
provider: "openai",
});
const resp = await create({ model: "gpt-4o-mini", messages: [/* … */] }); // metered; resp untouched
// Or meter a response you already have.
await meter.recordResponse({ workflowId: "cart-parse", response: resp, provider: "openai" });Honesty: tokens are read verbatim from the real response (extractUsage) —
cache reads are billed separately, never double-counted as fresh input — and a
metering failure is fail-safe (your call still returns / still throws). A failed
provider call is still recorded with status:"error" so it appears in the chain.
Report your evals (promptfoo)
Your eval suite already grades pass/fail — that is the outcome signal Margin's cost-/value-per-outcome math divides by. If you run promptfoo, point this adapter at the file it already writes; no hand-written POST:
promptfoo eval -o results.jsonimport { readFile } from "node:fs/promises";
import { MarginMeter, promptfoo } from "margin-meter";
const meter = new MarginMeter(); // MARGIN_INGEST_URL/KEY from env
const data = JSON.parse(await readFile("results.json", "utf8"));
const { recorded, skipped, errors } = await promptfoo.recordResults(meter, data, "my-eval-suite");One promptfoo result becomes one outcome. success is the pass/fail; score rides
through when it is in [0, 1]. The grade's provenance is derived from the assert
types, never assumed — a code assert (contains, equals, regex, …) records
ground_truth/code, any model-graded assert (llm-rubric, factuality,
similar, …) records llm_judge/judge, and the weaker honest claim always wins,
so a rubric grade is never reported as ground truth. Tag a case with
vars.margin_value: <usd> (or testCase.metadata.margin_value) to report the
outcome's worth — the value-per-outcome numerator; absent is the honest
cost-per-outcome default, never 0. A row with no boolean success is not a
verdict and is skipped rather than invented. Same contract, field for field, as the
Python SDK's margin_meter.promptfoo.
Run Braintrust instead? Same shape, one difference — Braintrust scores are continuous, so a pass is derived rather than intrinsic:
import { MarginMeter, braintrust } from "margin-meter";
const meter = new MarginMeter();
const data = JSON.parse(await readFile("results.json", "utf8"));
// a case passes when its mean score clears the threshold …
await braintrust.recordResults(meter, data, "my-suite", { threshold: 0.7 });
// … or, with no threshold, only cases whose scorers are all binary (0/1) are
// recorded; a continuous-score case is skipped, never given a fabricated verdict.qualityScore is the mean of the case's per-scorer scores; the grade's provenance is
derived from the scorer names (a model autoeval like Factuality → llm_judge/judge,
a code scorer like ExactMatch → ground_truth/code). Tag metadata.margin_value
for rung 8, and metadata.margin_judge_model to record which model judged (a bare
model field is the generation model under test, not the judge, so it is never used).
Your unit tests are already an eval harness — a passing test is a code-graded, ground-truth pass/fail. Point the adapter at the JSON your test runner already writes and every test becomes an outcome, no hand-written POST. One adapter covers both vitest and jest (they emit the same jest-compatible JSON):
import { MarginMeter, vitest } from "margin-meter";
// vitest run --reporter=json --outputFile=results.json (or: jest --json --outputFile=results.json)
const meter = new MarginMeter();
const data = JSON.parse(await readFile("results.json", "utf8"));
const { recorded, skipped } = await vitest.recordResults(meter, data, "my-test-suite");A passed assertion records as ground_truth/code with qualityScore 1.0, a failed
one 0.0; a skipped/todo/pending test is not a verdict and is skipped, never
fabricated. The link traces back to the test's fullName.
The JSON file carries no per-test value, so that path is cost-per-outcome only. To attach a
value (value-per-outcome, the way pytest uses @pytest.mark.margin_value), use the runtime
reporter instead — it runs inside vitest and reads task.meta.marginValue:
// vitest.config.ts
import { vitest as m } from "margin-meter";
export default {
test: { reporters: ["default", new m.MarginVitestReporter({ workflowId: "my-test-suite" })] },
};
// in a test — tag its business value (optional; absent = cost-per-outcome)
import { getCurrentTest } from "vitest";
test("resolves the ticket", () => {
getCurrentTest().meta.marginValue = 120; // this outcome is worth $120
// ...
});The reporter is a no-op unless a workflowId (or MARGIN_VITEST_WORKFLOW) is set, and it never
throws into your test run — a delivery fault warns and is counted.
API
| Method | Emits to | Notes |
| --- | --- | --- |
| recordCall(input) | POST /api/ingest/calls | cost computed from the pricing table when costUsd omitted |
| recordOutcome(input) | POST /api/ingest/outcomes | qualityMethod records HOW the score was graded |
| measure(input, fn) | recordCall on completion | times latency; status:"error" + rethrow if fn throws; returns fn's value, emit result on m.result |
| recordResponse(input) | recordCall | meters straight off a provider response — no manual tokens |
| wrap(fn, opts) | recordCall per call | returns an auto-metered drop-in for any LLM call |
| instrumentOpenAI / instrumentAnthropic / instrumentGemini | — | monkeypatch a client instance's create method |
| stage(op, opts?, fn) | — | attributes every call inside fn to one pipeline step under a shared run id; opens a chain node; returns fn's value |
| span(opts, fn) | — | marks the calls inside fn as one node of an agent tree, with the orchestration shape; nest to build the tree |
| currentStage() / currentSpan() | — | the active step / tree node, or undefined outside any |
| deliveryStats() | — | what this meter delivered against what it tried to (see below) |
Every method resolves to an IngestResult { ok, statusCode, body?, error?, reason? }.
ok is true only on HTTP 200; body holds call_id/outcome_id + source.
reason is the failure as a stable bucket (no_key, transport,
http_<status>) — read that rather than parsing error, whose wording is prose
and will move.
Batching (what it does to your process)
Calls are buffered and flushed in batches — 100 records or 10 seconds,
whichever comes first. recordCall() pushes onto a bounded queue and resolves;
the POST happens off your await chain, driven by an unref'd timer that never
holds your process open.
Why it works this way: at 1M metered calls/day, one call per HTTP request is 30M serverless invocations a month and the same traffic batched at 100 is 300K. That 100x is Margin's cost, not yours, which is exactly why it belongs in the SDK rather than the server — only the client can decide not to make the request.
| knob | default | env override |
| --- | ---: | --- |
| records per flush | 100 | MARGIN_BATCH_MAX_RECORDS |
| seconds between flushes | 10 | MARGIN_BATCH_FLUSH_SECONDS |
| queued records (memory ceiling) | 10,000 | MARGIN_BATCH_MAX_QUEUE |
What you are guaranteed: metering never blocks your call path, never throws into it, and cannot grow without bound in your process.
What it costs you, stated plainly:
recordCall()resolves to{ ok: true, body: { queued: true } }— accepted into the buffer, not written to the ledger.deliveryStats()is where you learn what landed.- A hard kill loses at most 100 records or 10 seconds of them.
beforeExitcovers a normal exit; no in-memory buffer survivesSIGKILL. - If Margin is unreachable long enough to fill the queue, the newest records
are dropped, and every drop is counted under
queue_full. - Delivery is at-least-once. Every record carries an
event_idand the server is idempotent on it, so a redelivery never double-counts your spend.
await meter.flush() resolves once everything queued has been delivered or
counted. await meter.close() drains and stops the timer. raiseOnError: true
skips the buffer entirely and posts at the call site, because strict mode's whole
meaning is "throw where I called you".
Full contract, including the arithmetic behind every default:
docs/reference/SDK_BATCHING.md.
Knowing what you dropped
Metering is fail-safe: a failed emit resolves to ok: false and your call carries
on. That is the right trade — telemetry should never take your agent down — but it
means a lost row looks, from the ledger, exactly like a call that never happened.
Cost per outcome then reads cheaper than your bill.
So the meter counts its own losses:
const s = meter.deliveryStats();
console.log(s.attempted, s.delivered, s.dropped, s.dropRate);
console.log(s.droppedByReason); // { http_502: 3, transport: 1 }The Python SDK's delivery_stats() reports two more fields, deduped and
unmeasured. Neither applies here: this client has no file sink to lose buffered
rows in and no cross-hook de-dup to suppress a second hook, so every emit it makes
goes through one door and is counted there.
Testing against the ingest contract (no network)
The network boundary is a single post(path, { json, headers }) protocol, so a
test can inject a fake that mimics the ingest API's documented 200/401/422/429
responses:
const meter = new MarginMeter({ apiKey: "mgk_…", transport: fakeIngest, raiseOnError: true });See test/meter.test.mjs. Run with npm test
(node --experimental-strip-types --test test/*.test.mjs — zero deps).
Rate + validation bounds
Ingest is auth'd, validated, and rate-bounded server-side: bad key → 401,
malformed/implausible row → 422, full rolling per-project window → 429. In
fail-safe mode these come back as { ok: false, statusCode }.
