@inferometer/probe

probe measures the performance of LLM inference providers (any OpenAI-compatible streaming chat API) and optionally contributes the results to the Inferometer board. It records time to first token (TTFT), total latency, output tokens per second, inter-token latency (ITL), token counts (including the reasoning slice when the provider reports it), finish reason, HTTP status, and classified errors, then prints a JSON report.

The published package is a single, dependency-free bundle that runs on Node >= 18. No clone, no Bun, no workspace install required — everything (the provider catalog, zod, yaml, and the shared schemas) is inlined at build time.

Quick start (no clone needed)

# Zero-install: run straight from npm (measures with YOUR keys)
npx @inferometer/probe --local

# Or install globally for a `probe` command on your PATH
npm i -g @inferometer/probe
inferometer register          # create a Inferometer account (username/password, no email)
inferometer login             # or, if you already have an account
inferometer                   # contribute mode: measure every provider you have a key for
inferometer --local           # no account? measure + present only, never upload

Add your provider keys as environment variables first (the catalog names which variable each provider uses, e.g. CROFAI_API_KEY):

export CROFAI_API_KEY=***
inferometer --local           # measures every provider whose key is set

Usage

inferometer [--provider <name> --model <id>] [--config <path>] [--region <tag>]
      [--concurrency <n>] [--yes | --no-upload] [--local]
inferometer register
inferometer login
inferometer logout

• Contribute mode (no --provider): measures every catalog pair you have an API key for, with bounded concurrency (default 4, --concurrency <n>), and prints JSONL (one compact report per line) to stdout.
• Targeted mode (--provider + --model): measures exactly one pair and prints a pretty JSON report.
• --local: measure and present only — no login, no upload, no backend contact.
• --yes / --no-upload: upload overrides for automation; without either, an interactive run prompts Upload N report(s)? [y/N] (default no), and a non-interactive run never uploads.
• --config <path>: override the bundled catalog with your own YAML.
• --region <tag>: the location tag stored with the report. Omitted, it is community in every mode. Each report also records concurrency (the effective --concurrency; 1 in targeted mode), because parallel runs share one uplink and can depress each other's timings.

Run inferometer --help for the full reference.

What a report contains

One ProbeReport per provider×model run: the resolved settings, traceability hashes, every timed request, and the aggregated summary. Points worth knowing before you read the numbers:

• output_tokens is the provider's completion count with reasoning tokens INCLUDED; reasoning_tokens reports that thinking slice when the provider counts it (usage.completion_tokens_details.reasoning_tokens, or a flat usage.reasoning_tokens), and is null otherwise.
• output_tokens_per_sec divides only the VISIBLE tokens by the visible window (first → last visible token): output_tokens - reasoning_tokens when the count is known, otherwise the ~4-chars/token estimate of the visible text when reasoning was streamed uncounted. overall_tokens_per_sec keeps ALL completion tokens, because its window (dispatch → end of stream) includes the thinking time.
• itl_ms samples are inter-CHUNK arrival gaps between consecutive visible-content deltas, not true per-token gaps. When several deltas arrive in one network read — routine with SSE, where gateways batch frames — the gaps between them are near zero and the real wait appears as one large gap. ITL describes a provider's flush pattern as much as its speed, and compares only between providers that stream the same way. Reasoning deltas between two content deltas land inside the gap.
• A 200 whose stream carries no visible content token is an error (empty_response), not a success: http_status 200, output_tokens null.
• A timeout or dropped stream keeps what it measured (ttft_ms, first_token_ms, reasoning_duration_ms, itl_ms); total_ms stays null and error rows never enter the summary aggregates.
• Every field added after the first release is optional, so reports from older probes still validate.

Exit codes

| Code | Meaning | |------|---------| | 0 | run completed (even if some/all measured requests failed) | | 1 | usage error or unexpected internal error | | 2 | invalid or missing catalog | | 3 | missing selection (provider or model not in catalog) | | 4 | missing API key (referenced env var unset; or no keys in contribute mode) | | 5 | authentication failure (login rejected, or session rejected on upload) |

The backend URL comes from INFEROMETER_API_URL (default the production Inferometer backend; set http://localhost:8090 for local dev). Sessions are cached at ~/.config/inferometer/session.json.

Develop from source

The probe lives in the inferencecompare monorepo (Bun + TypeScript). For contributors to the repo:

git clone <repo> && cd inferencecompare
bun install
bun run probe -- --local        # runs the source shim (packages/probe/src/bin.ts) under Bun
bun test                        # 207 tests

packages/probe/src/bin.ts is the single entry point (dev, npm bin, and npx all run it). src/cli.ts is an import-safe module exporting main(); the bundled dist/bin.js is produced by:

cd packages/probe
bun run build     # bun build src/bin.ts --target=node --format=esm --minify, + shebang

The build inlines the YAML catalog, package.json (the single version source for both the report's script_version and the HTTP user agent), and all dependencies into one ESM file whose only external imports are node:* builtins, so the published package has zero runtime dependencies.

Releasing

Publishing is automated by .github/workflows/publish-probe.yml, which runs on a tag push matching probe-v*. To cut a release:

# 1. Ensure packages/probe/package.json "version" is the new version (e.g. 0.3.0).
#    That one field feeds the bundle, the report, and the user agent.
# 2. Tag and push:
git tag probe-v0.3.0
git push origin probe-v0.3.0

The workflow checks out, sets up Bun, runs bun install && bun run build && bun test in packages/probe, then npm publish --access public, and creates a GitHub Release for the tag. Publishing uses npm trusted publishing over OIDC (the job requests id-token: write and npm verifies it): there is no npm token in the repo or its secrets. The trust relation on npmjs.com names this workflow file, so renaming the file breaks publishing until npmjs.com is updated.