@jiajun0413/pi-provider-manager

Roundrobin failover engine for the pi coding agent.

Registers a virtual roundrobin provider that pools multiple real model candidates behind one virtual model, with sticky failover, idle-timeout guards, and per-candidate cooldown.

Attribution

This package is a fork and rebuild of @arcaneorion/[email protected] (upstream npm package; no public Git repo). The roundrobin failover engine architecture is the original author's work, used here under MIT.

This fork fixes a mid-stream idle-timeout bug (a candidate stream that starts then goes silent no longer hangs forever), hardens failover against unstable relays (upstream aborted is now a cooldown-able failure, not a turn-killing cancel; auth lookup is bounded by the idle guard), and strips the visual config panel and on-disk health store, keeping only the failover engine.

Install

pi install npm:@jiajun0413/pi-provider-manager

Restart pi. (pi loads .ts extensions via jiti — no build step.)

Features

• Roundrobin failover — one virtual roundrobin provider pools multiple provider/model candidates. Sticky strategy with automatic failover on first-response timeout or stream error. Each preset becomes a selectable virtual model.
• Mid-stream idle timeout — a candidate stream that starts then goes silent errors out after timeoutMs instead of hanging forever (prevents ghost subagent sessions).
• No mid-stream replay — once content or tool calls have been emitted, an error is terminal rather than replayed on another provider, to avoid duplicating output.
• Upstream-aborted is a failure, not a cancel — a relay that drops the socket / cancels upstream and surfaces stopReason:"aborted" is cooled down and the pool rotates, instead of killing the whole turn. Only a real user abort (the request's own AbortSignal) short-circuits.
• Auth-bounded idle guard — API-key/header resolution races against the same timeoutMs + attempt abort as stream chunks, so a hung key lookup can't stall the turn.
• Entry-time model population — virtual models register at load time with placeholder candidates so they appear in /model and static lists (pi-web's /api/models) before a session starts; session_start rebuilds with the real registry.
• virtualModel inheritance — the virtualModel block is optional; omit it and metadata is inherited from the group's first candidate (see Inheritance).

Usage

1. Create ~/.pi/agent/roundrobin/config.json (the "default" group):

   {
     "virtualModel": {
       "name": "GLM-5.2",
       "reasoning": true,
       "input": ["text"],
       "contextWindow": 1000000,
       "maxTokens": 128000,
       "thinkingLevelMap": { "max": "max" },
       "compat": { "thinkingFormat": "deepseek" }
     },
     "candidates": [
       { "provider": "k", "model": "glm-5.2" },
       { "provider": "wg", "model": "glm-5.2" }
     ],
     "timeoutMs": 16000,
     "cooldownMs": 120000,
     "strategy": "sticky",
     "log": true
   }

2. (Optional) Add more groups as *.json files under presets/. Each file's stem (name without .json) becomes a roundrobin/<name> model.

3. In pi, use /model to select roundrobin/default (or any preset name).

4. Requests now flow through the failover engine. If the current candidate times out or errors before the stream starts, the next candidate is tried automatically. If every candidate fails, the engine waits for the earliest cooldown and retries from the preferred candidate — bounded to 3 passes before the turn ends with the last error.

A mid-stream error is returned immediately rather than replayed (content or tool calls may already have been emitted).

Sticky strategy — stays on the current candidate until it fails. After cooldown, returns to the preferred (first) candidate automatically.

Multi-preset routing

Each preset under ~/.pi/agent/roundrobin/ registers as an independent virtual model whose model id equals the preset file stem (the .json suffix is stripped). config.json is the reserved "default" group; every *.json file under presets/ is an additional group. In /model all virtual models appear at once — pick roundrobin/<name> to route to that group's pool. Health and currentIndex are isolated per preset.

Configuration

~/.pi/agent/roundrobin/config.json

| Field | Default | Meaning | |---|---|---| | virtualModel | (optional) | Virtual model metadata: name, reasoning, input, contextWindow, maxTokens, thinkingLevelMap, compat. id is forced to the group name. Omitted fields are inherited from the first candidate (see Inheritance); omit the whole block for homogeneous groups. | | candidates | [] | { provider, model }[] referencing entries in models.json. Unknown pairs are skipped (if all are invalid the group is enabled=false). | | timeoutMs | 30000 | Idle timeout per stream event — first byte and every mid-stream chunk race against this. | | cooldownMs | 60000 | How long a failed candidate is skipped before it can be retried. | | strategy | sticky | sticky / round-robin / primary. | | log | true | Append failover events to ~/.pi/agent/roundrobin/roundrobin.log. |

Presets

*.json files under ~/.pi/agent/roundrobin/presets/<name>.json — each is a standalone config (same shape as config.json) and becomes a roundrobin/<name> virtual model (id = <name>, the .json suffix stripped). bak.* files are skipped. A corrupt preset is skipped silently; it never breaks the engine.

Inheritance

The virtualModel block is optional. When you omit a field (or the whole block), the engine fills it from the group's first resolved candidate — the Model from modelRegistry.find(provider, model) in your models.json. Precedence is three-tier:

1. Explicit virtualModel.<field> in the preset wins.
2. Else inherit from the first resolved candidate's Model.
3. Else a built-in default.

A homogeneous group (multiple providers of the same model) needs no virtualModel at all:

{
  "candidates": [
    { "provider": "k", "model": "glm-5.2" },
    { "provider": "wg", "model": "glm-5.2" }
  ],
  "timeoutMs": 16000,
  "cooldownMs": 120000,
  "strategy": "sticky"
}

For heterogeneous groups (mixed models, e.g. a "power" pool of Grok + Claude + Deepseek), keep an explicit virtualModel to unify behavior across the pool (force a single compat.thinkingFormat, or a conservative contextWindow).

Inheritance happens at session_start, when the real model registry is available. Pre-session_start the engine registers with placeholder candidates so virtual models still show in /model and static lists; fields are then filled from real candidates once a session starts.

Architecture

index.ts                          ← entry: registers the roundrobin provider
extensions/
├── model-roundrobin.ts           ← failover engine (streamRoundRobin → tryCandidates)
└── rr-config.ts                  ← read config.json + presets/ → GroupConfig[]

Cooldown (failure tracking) is process-local in-memory — a failed candidate's cooldown resets when pi restarts. This is by design: cooldown is an in-process failover notion, not a durable statistic.

License

MIT