@modelcanary/sdk

Official ModelCanary SDK — safely roll out LLM model upgrades using signed config bundles, shadow-mode telemetry, and verified eval-override callbacks.

ModelCanary is a control plane for LLM model upgrades: you define aliases (chat-bot, summarizer) that map to concrete targets (openai/gpt-4o-mini, anthropic/claude-3-5-sonnet) per stage (production, canary, staging). Your application asks the SDK "what model should I use right now for chat-bot?" — the SDK answers from a cached, cryptographically-signed config bundle without ever blocking your hot path on a network call.

npm install @modelcanary/sdk
# or: pnpm add @modelcanary/sdk
# or: yarn add @modelcanary/sdk

Requires Node.js 18+ (uses node:crypto, node:fs, and global fetch). Works on Node, Bun, Deno (npm: specifier), and any serverless runtime with Node-compat (Vercel, AWS Lambda, Cloudflare Workers with nodejs_compat).

Quickstart

1. Resolve a model from an alias

import { createModelCanary } from "@modelcanary/sdk";

const canary = createModelCanary({
  appId: "app_abc123",
  environment: "production",
  mode: "hybrid",                                // local cache + background refresh
  apiUrl: "https://your-modelcanary.example.com/api",
  sdkKey: process.env.MODELCANARY_SDK_KEY!,     // mc_sdk_...
  publicKeys: process.env.MODELCANARY_PUBLIC_KEY!, // PEM
  refreshIntervalMs: 60_000,
});

// In your request handler:
const target = canary.resolve("chat-bot", { stage: "production" });
// → { provider: "openai", model: "gpt-4o-mini", capabilities: [...], configVersion: 42, ... }

const completion = await openai.chat.completions.create({
  model: target.model,
  messages: [...],
});

2. Send shadow-mode samples (production vs candidate side-by-side)

When you've already started running a candidate model alongside production traffic (either by calling both yourself or via your gateway), ship the side-by-side outputs to ModelCanary so it can score the upgrade automatically:

canary.shadow.recordSample({
  alias: "chat-bot",
  stage: "production",
  productionProvider: "openai",
  productionModel: "gpt-4o-mini",
  candidateProvider: "anthropic",
  candidateModel: "claude-3-5-sonnet",
  inputSnippet: "Summarize this doc...",        // optional, capped at 8 KiB server-side
  productionOutput: { text: prodAnswer },
  candidateOutput: { text: candidateAnswer },
  productionLatencyMs: 412,
  candidateLatencyMs: 538,
  metadata: { userId: req.user.id, route: "/chat" },
});

recordSample is fire-and-forget. Network failures are swallowed and reported via the shadow.dropped telemetry event — your production traffic is never blocked or impacted by ModelCanary availability.

3. Verify eval-override callbacks (architecture-callback runtime mode)

For apps that use runtime_mode: "architecture_callback", ModelCanary will POST signed eval requests to your endpoint, asking your own infrastructure to run the candidate model under test (so it sees your real prompts/RAG/tools). Use the middleware to verify the signature:

import { createEvalOverrideMiddleware } from "@modelcanary/sdk/middleware";

const middleware = createEvalOverrideMiddleware({
  secret: process.env.MODELCANARY_CALLBACK_SECRET!,
});

// In your callback handler:
app.post("/modelcanary/callback", express.raw({ type: "application/json" }), (req, res) => {
  const verified = middleware.verify({
    body: req.body.toString("utf8"),
    headers: req.headers,
  });

  // Pass the verified override into resolve() to force the candidate target:
  const target = canary.resolve(verified.alias, {
    stage: verified.stage,
    allowEvalOverride: true,
    override: verified,
  });

  // Run your normal pipeline against `target.model`...
});

Modes

| Mode | Behaviour | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | local | Bundle is loaded once from bundle / file / env. No network. Ideal for unit tests, CI fixtures, fully air-gapped deploys. | | remote | Local sources are tried as a warm-start cache, then a one-shot fetch against apiUrl runs at construction time. No background timer. Good for short-lived processes (Lambda, edge functions). | | hybrid | Local sources activate immediately (zero cost on the hot path), then a background timer polls apiUrl every refreshIntervalMs and activates a newer version if one shows up. Recommended for long-running servers. |

resolve() is always synchronous and never blocks on the network. A hybrid client that loses control-plane connectivity keeps serving traffic on the last good bundle until process restart.

Fail modes

• failMode: "closed" (default) — missing config / alias / stage throws. Choose this when you'd rather page than serve traffic on stale assumptions.
• failMode: "open-with-fallback" — the SDK falls back to fallbackBundle (a literal ConfigBundle you ship with the app) on any resolve failure. Choose this when "always serve traffic" outweighs "always serve the latest config".

Local bundle sources

The local source chain is tried in this order:

1. opts.bundle — in-memory ConfigBundle (highest priority)
2. process.env[opts.localBundleEnv ?? "MODELCANARY_BUNDLE_BASE64"] — base64-encoded JSON bundle (great for serverless deploys where the bundle is baked into env at build time)
3. opts.localBundlePath — a JSON bundle file on disk

Telemetry

createModelCanary({
  // ...
  telemetry: {
    enabled: true,
    emit: (event, payload) => {
      // event: "model.resolved" | "config.loaded" | "config.activated"
      //      | "config.refresh_failed" | "fallback.used" | "shadow.dropped"
      myMetrics.increment(`modelcanary.${event}`, payload);
    },
  },
});

Hook these into Datadog, Honeycomb, OpenTelemetry, etc.

Errors

All SDK errors extend ModelCanaryError:

| Error | Thrown when | | ------------------------------ | -------------------------------------------------------------------------- | | ConfigNotFoundError | local mode and no source produced a bundle (and no fallback configured). | | ConfigSignatureError | Bundle signature missing / invalid / expired / wrong app or env. | | AliasNotFoundError | resolve() for an alias not present in the active bundle. | | StageNotFoundError | resolve() with a stage not configured for the alias. | | TargetNotFoundError | Alias points at a target that isn't in the bundle (corrupted bundle). | | CapabilityMismatchError | Required capabilities are missing from the resolved target. | | RemoteConfigFetchError | Refresh failed (also surfaced via config.refresh_failed telemetry). | | EvalOverrideSignatureError | Callback middleware verification failed. |

import { AliasNotFoundError } from "@modelcanary/sdk/errors";

try {
  canary.resolve("nope");
} catch (err) {
  if (err instanceof AliasNotFoundError) { /* ... */ }
}

Browser / edge usage

The shadow recorder is fully isomorphic and only needs fetch:

import { createShadowRecorder } from "@modelcanary/sdk";

const shadow = createShadowRecorder({
  appId: "app_abc123",
  apiUrl: "https://your-modelcanary.example.com/api",
  sdkKey: "mc_sdk_...",
});
shadow.recordSample({ /* ... */ });

The createModelCanary client requires node:fs / node:crypto. On Cloudflare Workers, set compatibility_flags = ["nodejs_compat"]. On Deno, the npm:@modelcanary/sdk specifier handles compatibility automatically.

Graceful shutdown

process.on("SIGTERM", async () => {
  canary.stop();              // stop the hybrid background timer
  await canary.shadow.flush(); // drain in-flight shadow POSTs
  process.exit(0);
});

Compatibility

• TypeScript: ships native .d.ts. Strict-mode safe.
• Module formats: dual ESM (import) + CommonJS (require).
• Frameworks: framework-agnostic — works with Express, Fastify, Hono, Next.js, Remix, Nuxt, NestJS, vanilla Node, Bun, Deno, Cloudflare Workers (with nodejs_compat), and Vercel / AWS / GCP serverless.

License

MIT © ModelCanary