batchwatch — TypeScript / JavaScript client

Client for batchwatch.dev: crowdsourced measurement of queue time on LLM batch APIs.

Batch endpoints cost 50% of the synchronous ones, but "completes within 24 hours" is impossible to plan around. batchwatch measures what the queue actually does and answers one question: should I use batch for this job?

No dependencies. fetch, node:fs, and nothing else.

Install

npm install batchwatch

On npm. Or vendor src/ — it is two files, and Node 22.18+/24 runs the .ts directly.

Two lines

import { Batchwatch } from 'batchwatch';

const bw = new Batchwatch({ token: 'bw_...' });   // falls back to $BATCHWATCH_TOKEN

// 1. before you submit — does this belong in the queue?
if (await bw.shouldBatch('gpt-5.6-sol', { maxWait: '15m' })) {
  const job = await openai.batches.create(...);
} else {
  const answer = await openai.chat.completions.create(...);
}

// 2. measure it, so the next person gets a better answer
const t = bw.track('gpt-5.6-sol', { inputTokens: 9720 });
const result = await waitFor(job);
t.done({ outputTokens: result.usage.completion_tokens });

measure() closes the measurement for you, including on the error path:

const result = await bw.measure('gpt-5.6-sol', { inputTokens: 9720 }, async (t) => {
  const r = await waitFor(job);
  t.done({ outputTokens: r.usage.completion_tokens });
  return r;
});

Where the runtime supports explicit resource management, using t = bw.track(...) closes it at the end of the block.

Get a key with no email and no card:

curl -X POST https://batchwatch.dev/v1/keys -d '{"label":"my pipeline"}'

The deadline guard — batch when you can, sync when you must

Batch is half the price, but a queue that misses your deadline can take down a product. bw.batch(...) gets you both: it runs your batch, watches the clock, and if the batch has not finished by your deadline it cancels it and runs your synchronous fallback instead — so your job always gets an answer, on time.

const job = bw.batch('gpt-5.6-sol', {
  deadline: '15m',
  onDeadline: () => openai.chat.completions.create(...),
});
await job.submit(() => openai.batches.create(...));
const result = await job.result();

You hand over two callables — the batch-create and the sync fallback — and the client runs them. It never sees or builds your provider payload; there is no field for it, exactly as with track(). If the batch finishes in time you get its result; if the deadline fires you get the fallback's result.

Every fallback is a measured prediction outcome: "batching would have missed — running sync was right." It goes down the same accuracy path shouldBatch() already feeds, so the server can score how often the guard was needed. Nothing new leaves the machine.

deadline speaks the same duration language as shouldBatch's maxWait — '15m', '6h', '30s', a bare number of seconds, or null for no guard. The defaults duck-type the OpenAI / Anthropic batch shape; a different provider passes poll / cancel / resultOf callables.

The poll loop is ours, not yours

result() owns the wait so you do not write the same sleep/backoff loop everyone else does. It polls with exponential backoff and jitter, never faster than a rate-limit floor (a naive one-second loop against a 24-hour job is 86,400 requests and an angry provider) and never slower than a ceiling. The first interval is informed by the model's own measured p50 — no reason to poll every five seconds against a model whose median queue time is forty minutes — and reading that p50 fails open: if batchwatch is unreachable, polling simply continues on the fixed fallback schedule.

The 24-hour expiry is a distinct terminal state, never silently a timeout: job.expired is true when the batch hit the cutoff, so you can tell "the queue was slow" from "the batch failed".

Tune the cadence if you need to; the defaults are sane:

const job = bw.batch('gpt-5.6-sol', {
  deadline: '6h',
  onDeadline: () => openai.chat.completions.create(...),
  pollBase: 30, pollFloor: 5, pollCeiling: 900,
});
await job.submit(() => openai.batches.create(...));
const result = await job.result();

Driving an async or worker loop yourself instead of awaiting result()? Step the same state machine one poll at a time:

import { PollState } from 'batchwatch';

const state = await job.pollOnce();        // RUNNING / DONE / EXPIRED / FAILED
if (state === PollState.RUNNING) {
  await sleep(job.nextInterval * 1000);    // backoff + jitter, already applied
}

Partial completion — landed, failed, expired

A batch of 20,000 requests is not binary: some land, some fail per-request, and some never return before the 24-hour expiry. job.split() separates the three, mapped back to your own objects by custom_id (never by index — provider ordering is not guaranteed):

const result = job.split(downloadedLines, everySubmittedId);
result.landed;        // count that came back clean
result.failed;        // count that failed per-request
result.expired;       // count still outstanding at the 24h cutoff
result.complete;      // true only if EVERYTHING landed — never a silent success
result.resultFor('my-request-42');   // your object for one id, mapped correctly

// retry only what failed — idempotent, so a second call submits nothing new
const child = job.retryFailed((failedIds) => openai.batches.create(...));

An expired job is reported to batchwatch as status="expired", which is recorded but kept out of the percentiles (only completed rows count) — so a slow queue neither pollutes p90 nor loses the "the queue was slow" signal. That measurement rule lives with the ingest contract on the server, under /v1/calls/complete → "Partial completion", not only here in the client.

It fails open, always

If batchwatch is down, slow, or broken, your job must not notice. That is the first requirement, ahead of collecting any data at all.

• track() and done() are synchronous and do no network I/O on your call path — submissions are fire-and-forget promises.
• Two-second timeout by default (AbortSignal.timeout, BATCHWATCH_TIMEOUT in seconds).
• Every batchwatch error is swallowed and passed to the optional onError callback. Nothing is logged unless you wire it up.
• shouldBatch() is the one call you await, because you want the answer. If it cannot answer you get your own fallback back — never a guess. The default is false, "run it synchronously": being wrong that way costs money, being wrong the other way blows a deadline.
• An exception from your own callback in measure() is recorded as failed and re-thrown untouched. We swallow our errors, never yours.

test/fail-open.test.ts proves it against a port nothing listens on and against a socket that accepts but never answers.

It never sends your content

No prompts, no completions, no system prompts, no tool calls, no file names. The request body is built from a fixed allowlist — provider, model, mode, endpoint, request count, token counts, timestamps, status — and everything else is dropped by clean() on the way out. There is no field to put text in.

test/no-content.test.ts asserts it on the bytes a real HTTP server received, with a positive control so the test cannot pass by the client simply sending nothing.

outputTokens defaults to null, never 0

You know your input tokens. You cannot know your output tokens before the model has answered. So the default is absence, not zero.

Zero is not a harmless placeholder: output costs five to six times as much as input, so a saving computed on zero output is systematically too low — measured at 3.4x too low on a real model — and nothing in the response would tell you. If you know a ceiling, pass maxTokens and the answer comes back labelled as a ceiling.

Note that done({ outputTokens: 0 }) really does send 0: zero is a measurement, absence is not.

Reading back your own calls

There is no per-id route: to see what the client actually sent, read your own contributions. Both calls REQUIRE a key and, unlike the measurement path, do NOT fail open — with no key they throw BatchwatchAuthError rather than silently returning nothing. The server's row comes back verbatim, keys and all.

const bw = new Batchwatch({ token: 'bw_...' });

// The measurements THIS key has contributed, newest page first.
const page = await bw.myCalls({ limit: 100 });
for (const c of page.calls) console.log(c.model, c.duration_s, c.status);
// page.next is a ready-made relative URL for the next page, or null on the last.

// This key's tier, contribution status and quota.
const me = await bw.keyStatus();
console.log(me.tier, me.contributing, me.quota.calls_left);

myCalls({ after, limit }) — the readback for track(). after is unix seconds (only rows whose started_server is strictly greater); limit is clamped server-side to 1–1000 (default 500). Both are optional and only sent when given. keyStatus() takes no arguments.

Spooling

When a measurement cannot be delivered, the completed record is appended to a JSONL file and replayed later through POST /v1/calls/complete. Losing measurements exactly when the network is bad means losing them exactly when they are most interesting.

• Default path: $BATCHWATCH_SPOOL, or batchwatch-spool.jsonl in the temp directory. spool: null turns it off.
• Replayed automatically, at most once a minute, right after a successful call — that is the moment we know the network is up. Call bw.flushSpool() yourself on shutdown if you want it drained on exit.
• Spooling requires a token. /v1/calls/complete takes your own timestamps, so it is closed to anonymous callers; without a key a spool file could never be sent, and writing one would just leak disk.
• The file is capped at 5 MB. Beyond that, measurements are dropped rather than filling your disk.
• A replayed measurement can arrive twice if the original PATCH reached the server but the response did not. Deliberate: a duplicate is visible in the dataset, a lost measurement is not.
• The file format is identical across the Python, TypeScript and .NET clients, so a spool written by one can be flushed by another.
• Where there is no filesystem (workers, browsers) the spool turns itself off instead of failing.

Configuration

| Option | Environment | Default | |---|---|---| | token | BATCHWATCH_TOKEN | none (anonymous) | | baseUrl | BATCHWATCH_URL | https://batchwatch.dev | | timeoutMs | BATCHWATCH_TIMEOUT (seconds) | 2000 ms | | spool | BATCHWATCH_SPOOL | <tempdir>/batchwatch-spool.jsonl | | enabled | — | true | | onError | — | no-op |

Runtimes

Runs on Node 20+, verified on Node 24.13. Deno runs through the same code paths — it reads the environment and node:fs the same way — so it is supported by construction; run the suite under Deno to confirm it in your own toolchain.

Tests

npm install        # only @types/node and typescript, both dev-only
npm test           # node --test, runs the .ts sources directly
npm run typecheck
npm run build      # emits dist/ with .d.ts

84 tests, no network beyond loopback. They start real HTTP servers on ephemeral ports rather than stubbing fetch: the thing under test is network behaviour, so the network should be in the test. The poll-loop tests use an injected clock, sleep and rng, so the backoff schedule is pinned exactly with no real sleeps.

Licence

MIT