smart-model-run

Benchmark-driven smart model selection and fallback execution for agent workloads.

Install

pnpm add smart-model-run

For local development from the-watch:

{
  "dependencies": {
    "smart-model-run": "file:../smart-model-run"
  }
}

Usage

import { smartRun } from "smart-model-run";

const result = await smartRun({
  budget: "cheap",
  ceiling: "mid",
  thinking: "low",
  prompt: "Review this diff for correctness issues.",
  tools: ["read", "grep", "find", "ls"],
  needs: ["tools", "correctness", "codeQuality"],
  cwd: process.cwd(),
  runner: async (input) => runYourAgent(input),
  fallbackSelectors: ["openai/gpt-4o-mini"],
});

The library fetches model benchmark metadata from aistupidlevel.info, caches it at ~/.pi/agent/model-rankings.json for 6 hours, and falls back to cached rankings on network failure.

Local-only mode

Use local: true when callers must stay on local providers (ollama/*, lmstudio/*, or local/*):

await smartRun({
  prompt,
  budget: "cheap",
  local: true,
  needs: ["correctness", "codeQuality"],
  modelRegistry,
  runner,
  thinking: "low",
  tools: ["read", "grep", "find", "ls"],
  cwd: process.cwd(),
});

Local mode prefers enumerable local models from the Pi model registry, ranks them for code/review work, skips remote benchmark fetching by default, and appends this fallback chain for brittle or non-enumerable local setups:

1. ollama/ornith:35b
2. ollama/qwen2.5-coder:32b
3. ollama/qwen2.5-coder:14b
4. ollama/ornith:9b
5. ollama/ornith:7b
6. ollama/qwen2.5-coder:7b
7. ollama/llama3.1:8b

Local runs also use a longer default timeout of 10 minutes unless timeoutMs is provided.

Core exports

• smartRun() — select ranked candidates, run them, and fallback on failure.
• pickModels() — rank models within a budget tier.
• pickModelsAcrossBudget() — escalate across allowed budget tiers when no model meets minMatch.
• getRankings() / forceRefreshRankings() — fetch and cache live benchmark rankings.
• formatSmartRunSummary() / formatRankings() — human-readable summaries.