Maestro AI

Dynamic model delegation for agentic coding harnesses. Maestro routes LLM calls across local Ollama models and cloud-hosted models (via LiteLLM) based on task difficulty, risk, tools, and context size.

v1.9.6 fixes /status probe staleness and adds litellm.gatewayUp plus richer tier/route metadata. v1.9.5 adds proxy model pin (--model, X-Maestro-Model). v1.9.4 retries proxy streams on context overflow (tools omitted once). v1.9.3 adds proxy tier pinning (--model-tier, X-Maestro-Model-Tier). v1.9.2 validates configured model ids against gateway /v1/models in probe, doctor, and proxy startup. v1.9.1 hardens the Claude Code proxy (tool omission, chitchat bypass, agentic forwarding). v1.9.0 adds workload/role hints and cache-aware session stickiness. v1.8 adds workflow progress. v1.7 verification hooks + cost/feedback truth. v1.6 optional local_fast classify. v1.5 evidence-based routing, harness profiles, premium pool, sticky sessions.

Quick start (new machine)

git clone https://github.com/David-J-Shibley/maestro-ai.git maestro-ai && cd maestro-ai
npm install && npm run build

maestro init                    # creates ~/.maestro-ai/, MCP config, checks models
# merge ~/.maestro-ai/mcp-config.json into Cursor MCP settings

ollama pull llama3.2:latest     # if init reports missing models
ollama pull qwen3:8b
maestro doctor
maestro route "summarize this README" --debug

Ollama only? maestro init --profile ollama-only — no LiteLLM required.

Cloud only? maestro init --profile cloud-only — LiteLLM + API keys, no Ollama.

Full setup guide: docs/SETUP.md

Demo / “why Maestro?” docs/DEMO.md · docs/COMPARISON.md

Vision & roadmap: docs/VISION.md

Technical write-up: docs/ARTICLE.md

Quick start (already configured)

npx maestro route "summarize this README" --debug
npx maestro ask "Rewrite this commit message to be clearer"

Model tiers

| Tier | Default model | Endpoint | Use case | |------|---------------|----------|----------| | local_fast | llama3.2:latest | Ollama :11434 | Short prompts, formatting, simple classification | | local_strong | glm (primary) / qwen3:8b (fallback) | LiteLLM :4000 → Ollama | Summarization, rewriting, extraction, simple edits | | hosted_oss | qwen3-coder-next (primary) / qwen3:8b (fallback) | LiteLLM :4000 → Ollama | Medium coding, debugging, refactoring | | premium | claude-sonnet-4-6 | LiteLLM :4000 | Hard tasks, architecture, high risk, tool-heavy |

Config profiles in config/ — edit ~/.maestro-ai/config.json after maestro init.

Prerequisites

| Profile | Needs | |---------|-------| | default | Ollama :11434 + LiteLLM :4000 + FEATHERLESS_API_KEY (+ AWS for premium) | | ollama-only | Ollama only | | cloud-only | LiteLLM + API keys |

npx maestro probe
npx maestro doctor
npx maestro stats --last 50
npx maestro analyze

Install options

# npm (published package)
npm install -g maestro-ai
npx maestro-ai init --profile ollama-only

# From source
git clone https://github.com/David-J-Shibley/maestro-ai.git
cd maestro-ai && npm install

See docs/ARTICLE.md for npm vs git clone, MCP setup, and programmatic API usage.

Examples

Browser-based capability demos live under examples/:

• docs/DEMO.md — start here for CLI + Claude Code proxy demo scripts
• docs/COMPARISON.md — Maestro vs LiteLLM / static agent routing
• examples/run-routing-demo.sh — dry-run routing demo (no backends)
• examples/demo.html — Claude Code capabilities overview
• examples/claude-code-demo.html — interactive Claude Code demo
• examples/delegate-subtask.sh — shell hook delegating a subtask to Maestro

Claude Code integration (MCP — recommended)

Cursor

Run maestro init, then merge ~/.maestro-ai/mcp-config.json into Cursor MCP settings.

For git clones, see cursor-mcp-config.json — init generates machine-specific paths.

Claude Code

maestro init
# Use node path from ~/.maestro-ai/mcp-config.json
claude mcp add maestro-ai -- node <path-to>/dist/mcp-server.js

MCP tools

| Tool | Purpose | |------|---------| | maestro_route | Analyze task → return tier/model (no LLM call). Always includes analysis, debug, probe, fallback_reason. | | maestro_ask | Route + execute LLM call (optional workflow) with routing report | | maestro_workflow | Multi-step orchestration (auto / critique / implement-test-fix / …) | | maestro_probe | Health-check each tier primary and fallback endpoints | | maestro_doctor | Infrastructure diagnostics (process, port, /v1/models, env vars) | | maestro_stats | Telemetry summary — tier mix, escalation rate, latency, cost (insights: true for routing analysis) | | maestro_analyze | Per-task routing insights, recommendations, learned-routing readiness | | maestro_feedback | Structured feedback: rating (1–5), accepted, optional note |

MCP response shape

Every maestro_route / maestro_ask response includes:

{
  "tier": "local_strong",
  "model": "qwen3:8b",
  "analysis": { "taskType": "...", "difficulty": "...", "riskLevel": "...", "signals": [] },
  "debug": ["rule: summarization → local_strong", "tier_fallback: litellm down → ollama qwen3:8b"],
  "probe": { "unavailable_tiers": ["hosted_oss"], "results": [] },
  "fallback_reason": "primary unavailable",
  "endpoint_source": "tier_fallback"
}

debug and probe are always present — not gated on debug: true.

Every response also includes explanation — a human-readable decision card (explanation.markdown, explanation.why[]) describing why this model was chosen.

After maestro_ask, when the evaluator runs, the card also includes a validation outcome:

Selected: local_fast
Validation: failed schema check
Escalated to: local_strong
Final result: passed
Why: local output was incomplete

Telemetry records include an attemptLog with per-attempt pass/fail and failed checks.

Routing modes (v0.7+)

Operator control plane — one flag constrains tier selection, escalation, and retries:

| Mode | Behavior | |------|----------| | balanced | Default (cost-aware heuristics + escalation) | | local-only | Cap at local_strong — no cloud | | cheapest | Prefer local tiers, nudge to lowest viable | | fastest | Favor local_fast, no same-tier retries | | best-quality | Bias toward premium / hosted OSS | | private | Localhost only, privacy policy enforced |

maestro route "summarize this" --mode cheapest --debug
maestro ask "refactor auth module" --mode local-only --json

MCP: pass mode on maestro_route / maestro_ask. Telemetry and maestro stats report per-mode success rates.

Guardrails (v0.8+)

Declarative safety layer in policy.json → guardrails:

| Guardrail | Behavior | |-----------|----------| | budget | Warn when session budget is low; cap/block when exhausted | | privacy | Block cloud tiers when sensitive keywords match | | latency | Use probe latency to prefer faster tiers within target_ms |

Guardrail actions appear in the decision card under Guardrails.

Learned routing prep (v0.9+)

Turn telemetry into routing insights before enabling automatic hints:

maestro analyze                    # all records — recommendations & findings
maestro analyze --last 100         # recent window only
maestro stats --last 50            # summary stats

MCP: maestro_analyze or maestro_stats with insights: true.

Opt-in hints in ~/.maestro-ai/config.json:

"routing": {
  "learnedRoutingHints": true,
  "learnedMinSamples": 5,
  "llmClassify": "shadow"
}

• learnedRoutingHints — nudge tier from telemetry when confidence is high enough.
• llmClassify — off (default) | shadow | on. When heuristics are uncertain, call local_fast for a tiny JSON classify. Shadow only logs disagreements (llm_classify_diff=… in signals / /status); on merges into routing with fail-soft (cannot solo-force premium).
• offlineLocalOnly — default true. When offline (no internet, or cloud tiers unreachable while local is up), force maxTier: local_strong. Disable with "offlineLocalOnly": false. See connectivity on GET /status.
• cacheAwareSticky — default true. Within a session, soft cloud→cloud downgrades stay on the last cloud tier (prefix cache). Easy local turns can still leave cloud. Set "cacheAwareSticky": false to disable.

Workload roles (v1.9+)

For multi-agent or long sessions, pass an explicit role so routing isn’t only per-prompt heuristics:

| Role | Effect | |------|--------| | orchestrator / research / critic | Floor at hosted_oss | | coder | Floor local_strong (or hosted_oss when hard/high-risk) | | formatter | Cap at local_strong | | extractor | Prefer local → hosted range |

CLI: maestro ask "…" --workload orchestrator --session-id s1
MCP: workload on maestro_route / maestro_ask.

Workflow orchestration (v1.0+)

Maestro chooses an execution strategy, not just a model:

| Pattern | Use case | |---------|----------| | auto | Default — planner picks single-shot or multi-step | | critique | Draft → critique → revise (writing, docs) | | implement-test-fix | Code → tests → build validation → fix | | parallel-synthesis | Parallel workers → merge (compare, research) | | plan-execute-validate | Plan → implement → validate (complex tasks) | | extract | Extract → normalize → validate JSON |

maestro ask "build auth middleware" --workflow implement-test-fix \
  --run-tests "npm test" --run-build "npm run build" --debug
# stderr: [maestro] 1/6 Planner → running … → passed
maestro ask "compare these approaches" --workflow parallel-synthesis
maestro ask "review this RFC" --workflow critique --dry-run-workflow
maestro ask "..." --workflow auto --progress-json   # NDJSON on stderr

# After a call, feed structured feedback into learned routing:
maestro feedback <telemetry-id> --rating 4 --accepted --note "good enough"
maestro stats --last 50   # includes savings vs always-premium

Programmatic API:

import { runWorkflow, dryRunWorkflow } from "maestro-ai";

const result = await runWorkflow({
  messages: [{ role: "user", content: "Implement caching layer" }],
  workflow: "auto",
  mode: "balanced",
});
console.log(result.report.markdown);

Routing policy

Declarative rules in ~/.maestro-ai/policy.json (copied on maestro init):

• Task-type → tier overrides (e.g. architecture → premium)
• Privacy keywords → cap to local tiers
• Sensitive high-risk code → stay on localhost

See docs/VISION.md for the orchestration roadmap.

Pass once per chat session via MCP or CLI:

| Field | Effect | |-------|--------| | session_id | Correlate calls for budget tracking and sticky/cache-aware routing | | max_tier | Cap spend — never route above this tier | | budget_usd | Enforced — caps tier selection and blocks escalation when exhausted | | always_prefer_local | Prefer local_fast / local_strong when rules allow |

CLAUDE.md instructions

## Maestro AI (MCP)

Use Maestro MCP tools to offload cheap subtasks:

- **maestro_route** — check which tier a task needs before handling it yourself
- **maestro_ask** — delegate summarize/rewrite/extract/format/classify subtasks

Pass the literal task in `prompt`, not a meta routing description.
Keep complex coding, architecture, multi-file edits, and tool-heavy work in your own session.

CLI

maestro init [--profile ollama-only]   # first-time setup
maestro ask "<task>" --json
maestro route "<task>" --debug
maestro doctor
maestro stats --last 50
maestro analyze
maestro proxy --port 4100 --max-tier hosted_oss

Transparent proxy (v1.1+)

Point Cursor or Claude Code at Maestro instead of calling MCP tools for every subtask. Maestro still routes underneath.

node dist/cli.js proxy --port 4100 --max-tier hosted_oss --prefer-local
# or after npm link / global install:
maestro proxy --port 4100 --max-tier hosted_oss --prefer-local
maestro proxy --port 4100 --model-tier local_strong   # pin all requests to local_strong
maestro proxy --port 4100 --model qwen3-4b            # pin all requests to qwen3-4b

| Client | Base URL | Notes | |--------|----------|--------| | Cursor / OpenAI SDK | http://127.0.0.1:4100/v1 | POST /v1/chat/completions | | Claude Code | http://127.0.0.1:4100 | No /v1 — Claude appends /v1/messages |

Claude Code (--settings file or env):

{
  "env": {
    "CLAUDE_CODE_USE_BEDROCK": "false",
    "ANTHROPIC_BASE_URL": "http://localhost:4100",
    "ANTHROPIC_AUTH_TOKEN": "maestro",
    "ANTHROPIC_MODEL": "maestro"
  }
}

claude --settings ~/claude-featherless-settings.json

Tips:

• Cap with --max-tier hosted_oss (or local_strong) so tool-heavy Claude Code prompts don’t escalate to Bedrock.
• Pin with --model-tier local_strong (or per-request header X-Maestro-Model-Tier: hosted_oss) to force a specific tier without changing the client model id.
• On context overflow (truncated=1, zero streamed text), the proxy retries once with tools omitted before erroring — logged as context_retry=1.
• Pin a specific model with --model qwen3-4b (or header X-Maestro-Model: glm) to skip routing while keeping proxy tool/sanitizer logic.
• GET /status — connectivity, per-tier effective models (post-probe), LiteLLM gatewayUp vs reachable, process/config hints, and recentRoutes with toolsOmitted, forceToolUse, truncated, contextRetry.
• The proxy echoes your requested model id (clients won’t reject glm / maestro); real model is in maestro.routed_model.
• Tool passthrough: For LiteLLM backends, Anthropic /v1/messages is forwarded natively (tools, tool_use, tool_result, betas) so Claude Code agent loops stay intact. OpenAI-compatible fallback still converts tools when needed.
• Streaming is live token SSE (OpenAI or Anthropic event shapes). Request logs go to stderr as [maestro-proxy] ….
• For long routes, the proxy opens the SSE stream immediately and sends Anthropic ping / SSE comment heartbeats so Claude Code does not idle-reset the connection.
• Non-stream fallbacks (stream:false, used after Claude Code stream idle timeout) get leading JSON whitespace keepalives every 5s so the request timeout does not fire while Maestro routes.
• Live token streaming — stream:true routes via a fast dry-run decision, then forwards upstream SSE deltas as they arrive (not a single buffered dump).

Programmatic API

import { routedLLMCall, dryRunRoute, routedLLMStream } from "maestro-ai";

const result = await routedLLMCall({
  messages: [{ role: "user", content: "Extract function names from this code." }],
  taskHints: { type: "extraction", quality: "balanced", risk: "low" },
  overrides: { session: { maxTier: "hosted_oss", alwaysPreferLocal: true } },
});

Harness adapters

• benchyRouteSubtask / benchyDelegate — Benchy sub-prompt delegation
• resolveMaestroModel — Vercel AI resolveModel() replacement
• examples/delegate-subtask.sh — shell hook for Claude Code / Benchy

Configuration

After maestro init: ~/.maestro-ai/config.json

Bundled profiles: config/default.config.json, config/ollama-only.config.json, config/cloud-only.config.json

Override via MAESTRO_CONFIG env var.

Telemetry: ~/.maestro-ai/telemetry.jsonl

LiteLLM starter: config/litellm-minimal.yaml → copied to ~/.maestro-ai/litellm.yaml on init.

Routing rules

Premium — hard, high risk, tool-heavy + code, long context, system architecture.

Hosted OSS — medium coding, debugging, refactoring.

Local strong — summarization, rewriting, extraction, non-trivial edits.

Local fast — simple tasks, formatting, HTML/UI demos.

LiteLLM down? local_strong and hosted_oss fall back to Ollama qwen3:8b.

Tests

npm test

Migration from model-router

| Old | New | |-----|-----| | model-router/ | maestro-ai/ | | npx model-router | npx maestro | | model_router_route | maestro_route | | MODEL_ROUTER_CONFIG | MAESTRO_CONFIG | | ~/.model-router/ | ~/.maestro-ai/ |