AI Runtime

A provider-agnostic AI runtime — point it at whatever AI providers you have, and it routes each request to the best available model. On top of that it adds an interactive terminal, memory, skills, tools, planning, orchestration, comparison, and learning.

npm install ai-runtime-engine

Naming: the product is AI Runtime, the npm package is ai-runtime-engine, and the CLI command is ai-runtime. In code you use Runtime (the high-level API) and AI (the low-level router).

Quick start

In the terminal (install globally, or use npx ai-runtime …):

npm install -g ai-runtime-engine
ai-runtime setup      # scaffolds config + .env.example (asks before each write)
ai-runtime doctor     # checks a provider is configured and reachable
ai-runtime            # opens the interactive terminal — just type what you want

In code:

import { Runtime } from 'ai-runtime-engine';

const runtime = await Runtime.load();                 // reads your config + .env
const r = await runtime.run({ input: 'Fix the failing tests' });
console.log(r.response?.text);

Try it offline (no API key): the built-in mock provider runs the whole pipeline with no credentials.

import { Runtime, MockProvider, makeModel } from 'ai-runtime-engine';

const runtime = new Runtime();
runtime.ai.registerProvider(new MockProvider({
  id: 'demo',
  models: [makeModel({ providerId: 'demo', id: 'demo-1', caps: { intelligence: ['reasoning'], output: ['text'] } })],
}));
const r = await runtime.run({ input: 'hello' });
console.log(r.response?.text, r.routing?.selected);   // includes the full routing decision

How it works

You describe what you want; the runtime picks which model runs it. A few ideas carry the whole design:

• Providers are replaceable; capabilities are durable. You route by what a task needs (reasoning, JSON, a big context window…), never by a provider name.
• One router. Every model call — chat, planning, comparison, a skill's own call — goes through the same router: filter (capability · health · privacy · cost) → score → pick → execute → validate → fall back.
• Modes decide what kind of work it is: chat, plan, execute, orchestrate, agent, compare, debug. auto (the default) detects it from your request.
• Runtime vs AI. Runtime is the high-level API you'll use; AI is the bare router underneath it.
• Evidence over claims. A step succeeds only with proof (a tool returned ok, a validator passed) — never because a model said "done."

What you can do

• Chat, plan, or run multi-step tasks — preview any run with --dry-run (or dryRun: true) to see the plan and what it would touch, with zero changes.
• Compare models — compare mode (or runtime.compare(...)) runs a task across several models and reports agreement, differences, and contradictions; a validator you supply lets evidence pick the winner.
• Remember facts — durable, scoped memory that's retrieved when relevant (or turn it off with stateless mode).
• Add your own skills — drop a manifest into .ai-runtime/skills/, or install a skill pack from npm.
• Steer routing — hard-exclude or soft-prefer providers/models via config, per-run, or env vars.
• Connect MCP servers — mcp: in the config brings a server's tools in as ordinary permission-gated tools, deny-by-default per server. ai-runtime mcp and /mcp show what is connected.
• Delegate to agents — a plan step can hand bounded work to an agent with its own narrowed catalog, permissions and budget, which reports back structured findings. /agents lists them; /agents stop stops one. Turn it on with runtime.agents.enabled.
• Survive a crash mid-run — agent progress is committed as it happens, so resuming finishes the job instead of redoing it. ai-runtime resume-execution <id>.
• Fan out many model tasks as one unit — runtime.executeGroup(...) runs N tasks with bounded concurrency, per-task timeout, retries, optional quorum, budgets, cancellation, and resume — every task through the one router. runtime.evaluate({ candidate, rubric }) judges an answer against a rubric. These are generic primitives: a Council, a research swarm, or a review swarm is built on them, and the runtime never learns those product concepts (3.2.0+).
• Everything that acts is permission-gated and workspace-jailed (see Safety).

What's new in 3.0

The 3.x arc made the runtime reason about what it can do, work with tools it didn't ship with, delegate bounded work, and survive being killed in the middle of it.

• Action capabilities — the runtime knows it can read_file or run_tests independently of which tool or skill provides it, and tells you what's missing when a goal needs something it hasn't got.
• MCP — a zero-dependency client over stdio and streamable HTTP. Server tools become ordinary tools, with per-server grants that are off until you say otherwise.
• Agents — a third kind of plan step. An agent gets an envelope narrowed from the parent's catalog and permissions, a call budget it cannot exceed, and an output contract its findings must satisfy. Approving a plan shows you that envelope, because approving a delegation blind is approving a blank cheque.
• Durable agent work — plan, inner steps, findings and spend are on disk before the next wave starts, so a kill -9 costs you the step in flight and nothing else.
• Auto-decomposition (opt-in) — a goal can delegate to read-shaped agents nobody configured. The roles and their objectives ship in-tree; nothing the model writes becomes an objective, a tool, or a permission.

Upgrading from 2.x needs no config change. One default moved: the capability catalog is now included in planning prompts — set runtime.capabilities.catalog: false to remove it.

What's new in 3.2

Generic multi-agent execution primitives. The runtime can now run many model tasks as one bounded, resumable unit — the mechanics a Council, a research swarm, a debate, or a review swarm all need — without learning any of those product concepts.

• executeGroup — bounded sliding-window concurrency, per-task timeout, bounded retry-with-backoff, optional quorum (with a caller-chosen continue vs cancel-remaining policy), budgets (calls, tokens, wall-clock — tighten-only), cancellation propagated to the in-flight provider call, and persistence/resume that never re-runs a completed task.
• getExecutionGroup / resumeExecutionGroup / cancelExecutionGroup — observe, resume, and cancel a group. Resume re-runs only the tasks that didn't complete, bound to their request by a content hash.
• evaluate({ candidate, rubric }) — a generic single-evaluation primitive: it fences the candidate and rubric as untrusted data, demands a closed-set verdict, and degrades to inconclusive rather than throwing.

Every task runs through the same router as a single call — there is no second selection engine and no provider SDK is called directly, so a group is never a weaker privilege boundary than one run. Group statuses are their own unions (RuntimeStatus is unchanged), events are metadata-only, and the whole subsystem is inert until a method is called. Config caps live under runtime.executionGroups; retention via AI_EXECUTION_GROUP_RETENTION_DAYS. It is purely additive — existing single-run paths are unchanged. See the worked example in docs/GUIDE.md and the guarantees in docs/contract.md §11.

Configuration

Three places, and only these:

| Path | What | In git? | |---|---|---| | .ai-runtime/config.yaml | Project config — settings + providers (env-var names). | ✅ commit it | | .env | The secret values for those env vars. | ❌ gitignored | | ~/.ai-runtime/ | Local state — conversations, memory, executions, cache. Override with AI_RUNTIME_HOME. | ❌ never in git |

A minimal config:

# .ai-runtime/config.yaml
runtime:
  defaultMode: auto
providers:
  - id: gemini
    kind: gemini
    apiKeyEnv: GEMINI_API_KEY     # the NAME of the env var — never the value
  - id: ollama                    # local, no key required
    kind: ollama

Optional env vars (all optional): AI_DEFAULT_MODE, AI_CONTEXT_MAX_TOKENS, AI_MAX_COST_USD, AI_MAX_CALLS, AI_MEMORY_ENABLED, AI_LEARNING_ENABLED, AI_EXCLUDE_PROVIDERS / AI_EXCLUDE_MODELS / AI_PREFER_PROVIDERS / AI_PREFER_MODELS, AI_*_RETENTION_DAYS, AI_RUNTIME_HOME. Run ai-runtime setup to scaffold everything and ai-runtime info to see it resolved.

Recommended providers: Google Gemini (GEMINI_API_KEY), Groq (GROQ_API_KEY), OpenRouter (OPENROUTER_API_KEY), and Ollama (local, no key). Cloud providers need a key; Ollama runs locally.

Safety

Everything read through tools, files, the web, or MCP is treated as data, never instructions. Filesystem access is jailed to your workspace; shell is allowlist-first with a destructive denylist; writes and network are off by default. Secrets are named by env var only and redacted from all output — the npm package contains only the runtime code, never your keys, memory, or local state.

Documentation

This package bundles fuller docs in its docs/ folder (open them after installing):

• docs/contract.md — the public contract: what the Runtime guarantees (status/error/cancellation/ resume/persistence/security), separated from implementation detail.
• docs/GUIDE.md — a hands-on, study-friendly guide: install → first run → how it works → worked examples, runnable offline.
• docs/router.md — the engine reference: config keys, the run() API and AI class, providers, routing internals.
• docs/architecture.md — how the router and Runtime layers fit together.
• docs/security.md — the full security model.

License

ISC.