modelrig

A capability-aware AI gateway. You declare a route — the shape of the answer you need and the set of models allowed to serve it — and ModelRig picks a model that can, validates what comes back, retries the right way when it doesn't, and records every attempt. Routes are code, failures are typed, and cost is capped by an envelope you set.

• Node ≥ 20
• Works standalone with just a provider key. The hosted console, bake-offs, and the model registry are an optional upgrade — see Signing up.

Install

npm i modelrig

Native dependency. ModelRig writes one telemetry row to a local SQLite file before each run returns, so a run never blocks on the network. That buffer is better-sqlite3, a native module — prebuilt binaries cover macOS and Linux on x64/arm64 (nearly everyone); an unusual platform will compile it at install time and needs a C++ toolchain. Node ≥ 20 is assumed.

modelrig is a normal dependency: your existing npm install (or pnpm / yarn) step installs it prebuilt — there is no separate "build modelrig" step. Three things to get right when you deploy:

• Run it on a Node runtime, not an Edge/Workers runtime. better-sqlite3 is a native module and loads on import, so an Edge runtime can't run it. A standard Node host — server, container, or a Node serverless function — works after npm install.
• If your deploy bundles server code, mark modelrig external so the native binary is loaded at runtime instead of bundled:
  • Next.js — serverExternalPackages: ['modelrig'] in next.config.js (Next 13/14: experimental.serverComponentsExternalPackages)
  • esbuild — --external:better-sqlite3 (and --external:modelrig)
  • webpack — add better-sqlite3 to externals
• Set MODELRIG_API_KEY on every service, worker, and cron that runs a rig — each process reports its own telemetry, so the key belongs on all of them, not just your web service.

(In a monorepo that vendors ModelRig as a workspace package — the SDK source rather than the published tarball — build that package before the app that imports it, so its dist/ types resolve. Consuming the published package needs none of this.)

Quickstart

This runs with only a provider key — no account, no signup.

1. Set a provider key (any one your route allows):

export GEMINI_API_KEY=...       # or OPENAI_API_KEY, ANTHROPIC_API_KEY, …

2. Define a route at modelrig/routes/example.support_summarize.yaml (the example the package ships, in modelrig/routes/):

route: example.support_summarize
version: 1
schema: ./schemas/support_summarize.schema.json  # JSON Schema (2020-12 recommended; draft-07 still accepted)
candidates:                                      # THE candidate set — nothing else can serve
  - provider: openai
    model: gpt-5.4-mini
  - provider: gemini
    model: gemini-3.1-flash-lite
  - provider: deepseek
    model: deepseek-chat
require: [schema_conformant]
prefer: [cost]
prompt:
  system: ./prompts/support_summarize.system.md  # {{var}} + {{#if capability.X}} blocks
  variables: [ticket, product, priorContext]
policy:
  retries: { content_invalid: 2, network: 4, capacity_shed: 3 }
  timeout_ms: 60000
  tier: flex
  json: native

3. Call it:

import { createRig, loadConfigFromEnv } from "modelrig";

const rig = createRig(loadConfigFromEnv());        // keys from GEMINI_API_KEY etc.
const result = await rig.run("example.support_summarize", {
  input: { ticket, product, priorContext },        // template variables
  tags: { run_id, step: "summarize" },
  budget: { envelope: run_id },                    // hard-stop cost envelope
});
// result.output  — schema-validated JSON (ajv), regardless of serving model
// result.meta    — servedCandidate, tokens (in/out/cached), cost, tiers,
//                  validation status, attempts by failure class
rig.close();

Data-governance routing (opt-in). Add zero_retention to a route's require, or pass zeroRetention: true on a single rig.run(...) call, to route only to zero-retention–designated endpoints. It is a hard, opt-in filter that fails closed: if no candidate is designated zero-retention, the run raises the no-eligible-candidate error rather than dispatching to a retaining endpoint. (Capability gating is on by default here — unlike OpenRouter's opt-in require_parameters.)

Request-time routing controls (opt-in). A single rig.run(...) call can narrow the candidate set: maxPriceUsdPerMTok drops any model over a price ceiling (unknown price + a ceiling → dropped, fail-closed), and providers: { only, ignore } restricts or excludes providers. maxPrice is a candidate filter (which models are eligible) — distinct from budget.envelope, which is a spend cap on the run's total cost; both compose, both fail closed on unpriced models.

Customer cache handle (opt-in). If your code already engineered provider caching, carry it through: rig.run(task, { input, cache: { key, provider } }) passes a customer-owned cache identifier (a Gemini cachedContents/<id>, or a prompt_cache_key value) verbatim to the matching-provider candidate. It wins over the route's cache: auto stamp, and a non-matching fallback dispatches without it. ModelRig carries the handle and prices the hits — it never creates, refreshes, or deletes the resource, so a long job needs your own TTL heartbeat (see the caching lifecycle doc). The Lane-B rig.runRaw({ provider, model, apiKey, …, cache: { key } }) seam takes the same field.

rig.runRaw also accepts provider knobs, each additive — absent ⇒ byte-identical dispatch: grounding: { mode: "native" } (native provider web grounding — gemini; grok fail-closed until CE-9b), serviceTier: "standard" | "flex" | "priority", reasoning: { level: "minimal" | "low" | "medium" | "high" } (reasoning effort¹), and responseFormat: "json_object" (syntactic-JSON forcing¹ — MIME-shaping, NOT schema enforcement, so meta.validated stays false). A declared directive a provider cannot honor FAILS CLOSED — a thrown RigFailureError with class invariant_violation, pre-dispatch (no meter, no provider call) — never a silent drop. Which provider honors which knob is one table — the runtime guard and the published provider × knob matrix are generated from the same RAW_KNOB_SUPPORT; ask it in code with rawKnobSupport(provider) (and GEMINI_THINKING_CAPABLE_MODELS is exported so you don't duplicate the thinking-model set). A grounded step that also passes a cache.key keeps its cached-token ratio: bake the googleSearch tool into the cache resource at creation (ModelRig sends cachedContent without tools, which the provider requires).

¹ Multi-provider reasoning and responseFormat ship in 0.4.0; on the published package reasoning/grounding are gemini-only.

runRaw — a BYOK passthrough (Lane B). When you don't need a route — no schema, no validation, no retries — call a provider/model directly with your own provider key:

const rig = createRig(loadConfigFromEnv());
const result = await rig.runRaw({
  provider: "gemini",
  model: "gemini-3.1-pro",
  apiKey: process.env.GEMINI_API_KEY!,   // runRaw is BYOK — pass your provider key
  systemPrompt: "",                       // one combined prompt? put it ALL in userPrompt, leave systemPrompt empty
  userPrompt: buildPrompt(input),
});

runRaw is BYOK — pass your provider key; ModelRig holds it for the one call and never stores it. It records one metered lane=raw telemetry row at the provider's list price (zero ModelRig margin) and throws a typed RigFailureError on failure, exactly like rig.run.

Called inside a rig.artifacts.run.start context, runRaw/runRawStream also records a ground-truth step in the run's graph (/runs/[id]) and carries the run's run_id tag — since modelrig 0.5.0, on success and on failure. Name it with stepKey (the same field RunOptions exposes on the routed lane; precedence stepKey > tags.step > an auto provider/model#n key), so a raw call and the artifacts you save around it group under one step.

The raw-lane timeoutMs default is tier-aware since 0.5.0 — standard/ priority 300 s, flex 900 s; an explicit timeoutMs always wins, and a timeout failure teaches the tier + budget in its fixHint. flex ⇒ set timeoutMs in minutes (default 900 s since 0.5.0). See the generated raw-lane defaults table.

runRaw-only construction (routesDir: null) — 0.4.0+. A Lane-B deployment that ONLY calls runRaw/runRawStream has no routes. Build it with routesDir: null and createRig skips route loading, the rig.yaml manifest, and the serveability gate entirely — so a stray route file it never calls (or a route whose provider key is absent in this deployment) can no longer down construction:

const rig = createRig({ ...loadConfigFromEnv(), routesDir: null });
await rig.runRaw({ provider, model, apiKey, systemPrompt, userPrompt });

loadConfigFromEnv() always yields a string routesDir, so the opt-out is a deliberate code-level choice — override it as above; there is no env var that disables routes, by design. On such a rig, rig.run and every other route-dependent surface (runStream, runBundle, bake-offs, watch, replay, verifySwap) throws a terminal invariant_violation naming the fix.

Every attempt is recorded in a local SQLite file (.modelrig/telemetry.db) tagged with your tags. Failures are typed — RigFailureError.failure.class is one of content_invalid, capacity_shed, network, refusal, cache_invalid, timeout, config_auth, budget_exhausted, invariant_violation, quality_rejected — with a separate retry budget per class, so a flaky network never spends your content-validation retries. What each class means, whether it retries, and how it backs off is on Routing & reliability.

Metadata capture ("capture now, analyze later")

Every inference records a versioned, metadata-safe attempt envelope (AttemptMeta@v1) on inferences.meta — in both the routed and the raw lane, from the same producer. It is the metadata the adapters already see and used to discard: finish reason, refusal, reasoning tokens, the provider-reported model snapshot and request id, coarse HTTP status / error code, sampling echo, tool names (declared kept, undeclared hashed) with counts, per-component cost, request and output hashes (never the content), request/response byte sizes, response char count, and a coarse context bucket.

It never contains free text. Every field is an enum, a number, a boolean, a hash, or an identifier. The producer drops any string that is over 128 chars or contains whitespace, and the hosted ingest endpoint re-applies the same rule as a backstop — so analysis later is possible without prompts or outputs ever being stored here. Building the envelope never blocks or changes a run: on any failure it is simply absent (meta is null).

Runs carry a companion RunMeta@v1 on runs.meta, set at run.start:

const h = rig.run.start({
  pipeline: "nightly-report",
  meta: { v: 1, trigger: "cron", actor: "scheduler", config_hash: "…" },
  // pipeline_sha auto-fills from git when you don't set it (never fails the run)
});

Reserved dimensions

Business dimensions live in tags. Five keys are well-known and become first-class group-bys in cost and coverage views — use them verbatim so your data lines up with the console's built-ins:

| key | meaning | |---|---| | subject | the customer/tenant the work is for — an opaque id, hashed if it looks like PII | | feature | the product feature the call serves | | user | the end user (an opaque id) | | session | the conversation/session the call belongs to | | cost_center | the team/budget to attribute spend to |

modelrig validate warns on near-misses — a declared rig.yaml dimension named customer, tenant, user_id, team, … is nudged toward the reserved key it resembles. They stay ordinary tags; nothing is required.

Runs — the standard path

Every pipeline execution is a run — the standard way ModelRig records what your workstream did, not an appendix. Wrap the execution in a run context, name one step per model-call family, and save the work products as artifacts; the console's Runs tab then shows the run → step → artifact chain you click through, and /projects groups it by your tags.

import { createRig, loadConfigFromEnv } from "modelrig";

const rig = createRig(loadConfigFromEnv());

// One run per pipeline execution. run.scope isolates concurrent runs; use
// run.start()/run.end() for one run at a time on a request path.
await rig.artifacts.run.scope(
  { pipeline: "support", episodeKey: ticketId },
  async () => {
    const result = await rig.run("example.support_summarize", {
      input: { ticket, product, priorContext },
      tags: { subject: customer, feature: "summarize" }, // run_id auto-stamped
      stepKey: "summarize",                              // names this step on /runs
    });
    // Save the work product — prompt / raw / parsed all save the same way.
    rig.artifacts.artifact.save(result.output, { name: "summary", type: "step_output" });
  },
);
rig.close();

• Tags: carry subject (what the run is about — an opaque id) and feature. Inside a run context the SDK stamps tags.run_id (the run's episodeKey, else its id) and tags.step (the current step key) on every attempt row of both lanes — a value you pass always wins, and outside a run context nothing is stamped, so the row is byte-identical.
• Raw lane too: a rig.runRaw call inside a run context records a ground-truth step (since 0.5.0), so grounded/cached BYOK pipelines fill the same graph.
• Acceptance: the run appears on /runs with its steps in order and the artifacts you saved attached. modelrig status prints runs recorded: N as the local mirror; modelrig validate warns if a rig records calls but never starts a run.

On by default under a control plane. The run/artifact namespace is on by default once a control plane is configured — set MODELRIG_API_KEY=rig_sk_… (or the self-host Supabase pair) and runs record automatically, metadata and hashes only. A pure local-only rig (no sink to ship to) stays off; force it on there with MODELRIG_ARTIFACTS=1, or opt out anywhere with MODELRIG_ARTIFACTS=0. With the namespace off it is inert — run.start is a no-op, artifact.save returns null, zero rows are written, and your pipeline behaves byte-for-byte as before. modelrig status prints the current posture.

Metadata and hashes only, this release. Content custody (holding the serialized bytes) is early access; today the value you save is hashed (sha256) and the bytes are discarded — the metadata row and its integrity hash are what persist. A zeroRetention run refuses every artifact fail-closed at the SDK gate.

The full artifact API — lineage (link), evaluations (evaluate), the reusable per-step seam, and run.scope vs run.start — is the instrumentation guide, shipped in this package so you can read it offline.

CLI

The package ships a modelrig binary:

npx modelrig init         # scan a repo for provider call sites, scaffold routes
npx modelrig bakeoff      # compare candidates on recorded inputs
npx modelrig watch        # keep the pricing/capability registry current
npx modelrig swap         # propose + actuate a candidate-set change (git-tracked)
npx modelrig samples      # inspect recorded telemetry samples

npx modelrig serve (a thin HTTP wrapper over rig.run) lives in a separate package, modelrig-server; the CLI tells you so if it isn't installed.

What you get by signing up

The gateway above is complete on its own. An account at app.modelrig.ai adds, as an upgrade — never a gate on the quickstart:

• a console for telemetry, cost, and route health across your runs,
• bake-offs and a maintained model registry (pricing + capabilities),
• hosted telemetry ingest — set MODELRIG_API_KEY=rig_sk_… and the exporter posts to the hosted API instead of writing only local SQLite.

Honest limits

• Prepaid balances and managed billing are not built yet. This release is the gateway and the self-host lane; the hosted control plane has one customer.
• 0.x means the shape may still move. Route and config contracts are stable enough to build on, not frozen.
• modelrig serve needs the modelrig-server package — it is not bundled here.
• What the ladder does and does not do. Fall-through is structural and infrastructural, plus an OPTIONAL SDK-lane qualityGate predicate you supply (a schema-valid output it rejects becomes quality_rejected); the route-declared hosted judge is not built yet, and the full boundary (raw-lane structure, latency ordering…) is generated and tripwire-tested on Routing & reliability.

License

Apache-2.0. See LICENSE.