copilot-api-router

A local aggregating router that fronts N independent copilot-api backends — each bound to its own GitHub account (concurrency quota + prompt cache) on its own VM — behind a single Anthropic/OpenAI-compatible interface.

The router spreads every request across healthy backends by least-outstanding in-flight count with a random tie-break — the single routing rule for both fresh sessions and continuations. It binds loopback only and has no inbound auth (single-user host assumption); it presents each backend's Proxy API key outbound. Backends are expected to be reachable on the router's own private network (same-subnet LAN, VPN, etc.) — deployment topology is the operator's responsibility.

Status: in progress. Implemented so far: config-driven backend list, loopback bind, least-outstanding load spread, raw-byte request forwarding with byte-for-byte SSE streaming, non-completion passthrough, a minimal GET /_router/status plane, periodic deep health checks (authenticated GET /v1/models) with ejection/recovery, a passive connection-level circuit breaker that cools a backend after repeated hard failures, request-path resource limits (max body size + bounded-concurrency backpressure), and client-specified backend pinning via a /backends/<alias>/… URL prefix.

v0.2.0 removed prefix-affinity session stickiness — every completion now routes purely by least-outstanding. Sticky routing kept a session on the backend that first served it to reuse its account-side prompt cache, but with only a handful of backends this starved parallel capacity when one session burst-fired (agentic tool loops). The trade-off is intentional: less cross- round cache reuse, more usable concurrency.

Install

Requires Bun installed locally (the bin's shebang is #!/usr/bin/env bun; the bundled binary calls Bun-specific APIs at runtime).

Install once and run:

npm install -g @dianshuv/copilot-api-router
copilot-api-router --config ./router.config.json --port 4141

Or run without installing:

bunx @dianshuv/copilot-api-router --config ./router.config.json --port 4141

Run from source

bun install
bun run src/main.ts --config ./router.config.json --port 4141

--host / ROUTER_HOST override the bind address; the default is 127.0.0.1 (an unconfigured instance is never exposed on the network).

Config

The backend list is config-driven — the count N is never hardcoded. Adding or removing an account is a config-only change:

{
  "backends": [
    { "url": "http://10.0.0.1:4141", "apiKey": "proxy-key-1", "name": "acct-a" },
    { "url": "http://10.0.0.2:4141", "apiKey": "proxy-key-2", "name": "acct-b" }
  ]
}

Each backend carries its own Proxy API key, which the router attaches outbound (Authorization: Bearer … / x-api-key).

Each backend may also carry an optional name — a stable alias used by client-specified backend pinning so a client's URL survives reordering backends[]. name must be a non-empty string, unique across backends[], and NOT all decimal digits (an all-digits name would collide with the numeric-index alias form).

Optional knobs

All optional; an unset knob falls back to the documented default below. Health cadence and the request hard-ceiling come from earlier slices; the resource limits (max body size + backpressure) are Issue 07; the passive connection-level circuit breaker (breaker*) is ADR 0002. v0.2.0 removed the endpoints, prefixTtlMs, and prefixMaxEntries knobs; a config file that still carries any of these will fail to load with a migration hint.

| Key | Default | Meaning | |---|---|---| | healthCheckIntervalMs | 20000 | Per-backend deep-probe cadence (authenticated GET /v1/models). | | healthCheckFailureThreshold | 3 | Consecutive failed probes that eject a backend. | | breakerFailureThreshold | 3 | Consecutive connection-level failures (refused/reset before first byte) on real traffic that trip a backend's cooldown. | | breakerCooldownMs | 30000 | How long (ms) a breaker-tripped backend is skipped by selection before auto-re-admission. | | requestHardTimeoutMs | 900000 | Hard ceiling (15 min) for one completion; never failed over. | | maxBodyBytes | 8388608 | Max inbound request body (8 MiB). A larger body is rejected with 413 and never buffered/forwarded. | | maxConcurrentRequests | 64 | Max completions streaming through the router at once (backpressure ceiling). | | maxQueueDepth | 256 | Max requests allowed to wait once the ceiling is reached; past this the router sheds load with 503 (set 0 to reject immediately, never queue). |

Logs

The router prints two lines per dispatched request — a dim [....] start line the moment the router commits to a backend for this request, then a [ OK ]/[FAIL] completion line when the request settles — so long-running streams are visible in flight instead of silent until they finish. Requests that fail before the router commits (413 oversize body, 503 no-healthy-backend, pin 404/503, queue-overflow) emit only the completion line — nothing was in flight, so there is no start line to pair.

14:23:00  [....]  /v1/messages          spread    #2/acct-a                   claude-opus-4
14:23:01  [ OK ]  /v1/messages          spread    #2/acct-a     200   1.243s  claude-opus-4
14:23:01  [....]  /v1/messages          spread    #0/acct-b                   claude-opus-4
14:23:02  [ OK ]  /v1/messages          spread    #0/acct-b     200   0.932s  claude-opus-4
14:23:05  [FAIL]  /v1/messages          none      --            503   0.004s
14:23:06  [....]  /v1/models            passthru  #1/acct-b
14:23:06  [ OK ]  /v1/models            passthru  #1/acct-b     200   0.088s

Columns are locked to the same fixed widths for both line variants so start and end lines align cell-for-cell: local time · [....]/[ OK ]/[FAIL] prefix · path (min-padded, never truncated) · routing outcome (spread / passthru / pinned / none) · backend (#index/alias if the backend was named in config, #index otherwise, or -- when no backend served) · HTTP status · total duration (seconds, 3 decimals) · client-requested model (from the request body, when a completion carried one). The start line leaves the status and duration columns blank — a start has neither yet. For a streamed completion the duration runs to the last byte (the whole request); for a passthrough or an error response it runs to the response headers. Map a backend index to its URL via GET /_router/status, which also reports per-backend inflight/health and the most recent routing decisions.

Failover (e.g. [....] spread #0 followed by [ OK ] spread #1) is diagnostic gold, not a bug: it means the first backend refused the connection and the router transparently retried on another. See ## Client-specified backend pinning below for the case where failover is intentionally suppressed.

When stdout is a TTY the router additionally keeps a sticky footer on the last line showing the live backpressure state — e.g. [....] 3 in flight, 12 queued requests — refreshed on every request completion. Piping the router's output (e.g. into a file or less) suppresses the footer so captured logs stay one pair per request; NO_COLOR=1 disables ANSI colouring, and FORCE_COLOR=1 enables it even when stdout is not a TTY (e.g. under systemd with StandardOutput=append:..., so tail -F piped to a terminal still renders colour).

Client-specified backend pinning

By default the router load-spreads every request across healthy backends by least-outstanding in-flight. Some callers want to bypass that — "for THIS request, use backend X". The router exposes a URL-prefix channel for that:

POST /backends/<alias>/v1/messages
POST /backends/<alias>/v1/chat/completions
POST /backends/<alias>/v1/responses
GET  /backends/<alias>/v1/models       # passthrough paths are pinned too

<alias> is either a backends[].name from config or the numeric index (/backends/0/…). The prefix is a router-side selector: it is stripped before the request is forwarded, so the upstream backend receives the original path (/v1/messages, not /backends/0/v1/messages). Everything else — request body, headers (minus router-rewritten auth), streaming — is unchanged.

Contract

| Situation | Response | |---|---| | Alias resolves to a healthy backend | Request forwarded to that backend; response streamed back | | Unknown alias (misspelt / deleted / OOB index) | 404 {"error":{"message":"unknown backend \"…\""}} | | Alias resolves but backend is unhealthy | 503 {"error":{"message":"pinned backend \"…\" is unhealthy"}} — never silently rerouted | | Pinned backend refuses connection at request time | 502 (upstream error) — no failover to any other backend | | /backends/<alias>/_router/status | 404 (router-owned paths cannot be pinned) |

Semantics

• Pin bypasses the default load-spread — the addressed backend is used unconditionally.
• Pinned requests are NOT failed over. The plain (unpinned) path retries a connection-level failure on another backend (per US-14); a pinned request surfaces the upstream error instead, because silently delivering the request to a different backend violates the pin's whole point.
• Pinned rounds show up on GET /_router/status as kind: "pinned" and in the CLI access log as the pinned column value, so an unbalanced-looking spread is diagnosable as "the client pinned it, not a router bug".
• Pin is session/process level, not per-request-within-a-CLI. The three supported CLIs (Claude Code / Codex / opencode) all bake base_url at startup and do not expose per-request overrides. Switch backend by starting a fresh CLI process (or, in Codex, by re-invoking codex exec with a different -c override).

Client configuration — point the CLI's base URL at the router's pin prefix:

# Claude Code — env or settings.json
export ANTHROPIC_BASE_URL=http://127.0.0.1:4141/backends/acct-a
claude

# Codex — ~/.codex/config.toml
# [model_providers.copilot-router]
# base_url = "http://127.0.0.1:4141/backends/acct-a/v1"

# opencode — opencode.json
# "provider": {
#   "gh-router": {
#     "npm": "@ai-sdk/openai-compatible",
#     "options": { "baseURL": "http://127.0.0.1:4141/backends/acct-a/v1" }
#   }
# }

Omit the prefix entirely (point at http://127.0.0.1:4141) to use the default load-spread path — pin is strictly opt-in.

Test

bun run test               # unit (pure logic: selection, config, bind-host)
bun run test:integration   # integration (spawn router + controllable fake backends)
bun run test:all           # both
bun run typecheck

Integration tests follow the copilot-api prior art: spawn the router as a child process with a readiness probe and ambient-env stripping, in front of controllable fake backends, and assert end-to-end behaviour (which backend was chosen, byte-for-byte stream fidelity, in-flight spread, credentials forwarded).

End-to-end (Makefile)

A Makefile wraps the router for end-to-end checks (it's what the e2e-verify workflow drives). The router is a loopback-only API service with no frontend, so verification is backend-only — curl the status plane.

make up          # start the router in the foreground (`make down` to stop)
make smoke       # probe a running router: GET /_router/status
make e2e         # one-shot: fake-backend stack + a real completion, then tear down
make down        # stop the router (or e2e-stack) started above

make up needs a config with at least one backend. If none exists it writes a placeholder to .tmp/router.config.json pointing at an unreachable backend: the router still binds and GET /_router/status confirms it is live, but completions cannot be served (no reachable backend). Point it at a reachable backend to exercise real forwarding (all knobs overridable):

make up CONFIG=./router.config.json   # also: PORT=4242 HOST=127.0.0.1

For real forwarding without a live backend, make e2e stands up N controllable fake backends (the integration-test fakes), points the router at them, fires one /v1/messages, and asserts the SSE hello from <backend> came back — then tears the whole stack down. make e2e-stack leaves that stack running in the foreground so you can poke it by hand (make down stops it):

make e2e BACKENDS=3      # assert real forwarding across 3 fake backends
make e2e-stack           # leave the stack running; curl it, then `make down`

Release

Bundled via tsdown to a single dist/main.mjs and published to npm. The package ships only the built artifact (dist/*.mjs) plus README.md / package.json — no source, tests, or sourcemaps.

NPM_TOKEN=<your-npm-token> bun run release

prepublishOnly gates the publish on typecheck + the full test suite (test:all); prepack (re)builds dist/. .npmrc reads NPM_TOKEN from the environment, so no token lives on disk.