smart-model-run

Benchmark-driven smart model selection and fallback execution for agent workloads.

Install

pnpm add smart-model-run

For local development from the-watch:

{
  "dependencies": {
    "smart-model-run": "file:../smart-model-run"
  }
}

Releasing

Publishing is automated. To cut a release:

1. Bump version in package.json as part of your PR.
2. Merge to main.
3. CI typechecks, tests, builds, and publishes to npm, then tags v<version> and opens a GitHub release.

Merges that do not change version skip the publish step and still pass green. The workflow authenticates via npm trusted publishing (GitHub Actions OIDC), so there is no npm token in repository secrets and every release carries a provenance attestation.

Usage

import { smartRun } from "smart-model-run";

const result = await smartRun({
  budget: "cheap",
  ceiling: "mid",
  thinking: "low",
  prompt: "Review this diff for correctness issues.",
  tools: ["read", "grep", "find", "ls"],
  needs: ["tools", "correctness", "codeQuality"],
  cwd: process.cwd(),
  runner: async (input) => runYourAgent(input),
  fallbackSelectors: ["openai/gpt-4o-mini"],
});

The library fetches model benchmark metadata from aistupidlevel.info, caches it at ~/.pi/agent/model-rankings.json for 6 hours, and falls back to cached rankings on network failure.

Local-only mode

Use local: true when callers must stay on local providers (ollama/*, lmstudio/*, or local/*):

await smartRun({
  prompt,
  budget: "cheap",
  local: true,
  needs: ["correctness", "codeQuality"],
  modelRegistry,
  runner,
  thinking: "low",
  tools: ["read", "grep", "find", "ls"],
  cwd: process.cwd(),
});

Local mode prefers enumerable local models from the Pi model registry, ranks them for code/review work, skips remote benchmark fetching by default, and appends this fallback chain for brittle or non-enumerable local setups:

1. ollama/ornith:35b
2. ollama/qwen2.5-coder:32b
3. ollama/qwen2.5-coder:14b
4. ollama/ornith:9b
5. ollama/ornith:7b
6. ollama/qwen2.5-coder:7b
7. ollama/llama3.1:8b

Local runs also use a longer default timeout of 10 minutes unless timeoutMs is provided.

Scoped model preference

Pass scopedModels when the caller has an explicit model scope. Available scoped models are tried first, in scope order, and global benchmark-ranked candidates are used only when no scoped model is available:

await smartRun({
  prompt,
  budget: "cheap",
  scopedModels: [
    { model: { provider: "provider-a", id: "fast-model" } },
  ],
  modelRegistry,
  runner,
  thinking: "low",
  tools: ["read"],
  cwd: process.cwd(),
});

Provider token exhaustion tracking

Use a shared providerTokenTracker to route future calls away from providers that report exhausted token or credit quota:

import { createProviderTokenTracker, smartRun } from "smart-model-run";

const providerTokenTracker = createProviderTokenTracker();

await smartRun({
  budget: "cheap",
  thinking: "low",
  prompt,
  tools: ["read"],
  cwd: process.cwd(),
  runner,
  providerTokenTracker,
});

When a run fails with quota-style errors such as out of tokens, insufficient_quota, or credit balance is too low, the provider is marked in the tracker. Later candidates from the same provider are skipped in the current run, and future smartRun() calls that reuse the same tracker skip that provider before invoking the runner.

Attempts include provider and outOfTokens fields so callers can display or persist token-exhaustion state themselves.

Core exports

• smartRun() — select ranked candidates, run them, and fallback on failure.
• pickModels() — rank models within a budget tier.
• pickModelsAcrossBudget() — escalate across allowed budget tiers when no model meets minMatch.
• getRankings() / forceRefreshRankings() — fetch and cache live benchmark rankings.
• createProviderTokenTracker() — build a reusable in-process token-exhaustion tracker for rerouting providers.
• formatSmartRunSummary() / formatRankings() — human-readable summaries.