@agentproto/llm-endpoint

v0.11.14

Published

3 days ago

@agentproto/llm-endpoint — a multi-surface LLM proxy gateway. Exposes Anthropic Messages, OpenAI Chat Completions, and OpenAI Responses API surfaces with transparent provider/model routing to Moonshot, OpenRouter, ZAI/Zhipu, Groq, xAI, and OpenAI. Handles

0High
0Medium
0Low

agentiknet

agentproto llm proxy anthropic openai-compat gateway open-standard

@agentproto/llm-endpoint

A lightweight multi-surface LLM proxy gateway. It exposes three API surfaces from a single port:

POST /v1/messages — Anthropic Messages compatibility. Claude-shaped model aliases are supported only through an explicit local compatibility pack.
POST /v1/chat/completions — OpenAI Chat Completions compatibility with transparent provider/model routing.
POST /v1/responses — OpenAI Responses API facade for Codex custom providers.

Requests are fanned out to upstream providers — Moonshot, OpenRouter, ZAI/Zhipu, Groq, xAI, direct OpenAI, Nebius AI Studio, a self-hosted "forge" server for fine-tunes, and any number of named local/LAN model servers (Ollama, llama-server, vLLM, …, configured from ~/.agentproto/llm-endpoints.json) — using provider-native model references. The proxy also handles Anthropic↔OpenAI schema translation, per-provider tool caps, orphaned-tool-call repair, and thinking-block stripping where needed.

Package: @agentproto/llm-endpoint
Entry: src/cli.ts → start() in src/index.ts
Default port: 18090 (override with LLM_ENDPOINT_PORT, or PORT)

Credentials this proxy handles

Read this before pointing the proxy at anything you care about. The package ships no credentials — every key below is one you supply — but it is a credential-forwarding component, so its handling rules are part of its contract.

Per-provider API keys resolve from one env var each (resolveSecretKeys), or from ~/.agentproto/providers.json. They are injected at proxy boot and never logged.
Anthropic subscription OAuth tokens (sk-ant-oat…) are supported as an alternative to an API key on the anthropic upstream. When one is resolved, the proxy sends Authorization: Bearer <oat> plus anthropic-beta: oauth-2025-04-20 and anthropic-version: 2023-06-01. This path is fail-closed: an OAuth credential is only ever sent to the anthropic upstream, and every other route 401s rather than leaking it to a third-party provider. Batch creation also refuses an OAuth token outright, because the Anthropic Batches API only accepts API keys.
A stored OAuth token works while it is valid. The proxy does not yet drive a self-refreshing source (source: "claude-code-oauth") to re-mint an expired one.
KeychainStore is macOS-only; elsewhere the proxy falls back to the per-provider env key.

Fronting a personal Claude subscription through a shared gateway may not be compatible with your Anthropic plan's terms — that is a question about your deployment, not about this package. If you expose the proxy beyond localhost, read Securing a public deployment first: the access gate is off by default.

Quick start (client config)

Point any Anthropic- or OpenAI-compatible client at the proxy:

| Setting | Value | | :--- | :--- | | Base URL | http://localhost:18090/v1 (local), or your own public origin + /v1 | | API key | The proxy injects the real upstream key server-side, so the client key is never your provider key. If the inbound access gate is enabled, send one of its tokens as the bearer; if it is not, any non-empty value passes. |

The client asks for a model by its provider-transparent reference (provider/model, e.g. moonshot/kimi-k2.7-code, openai/gpt-4.1); the proxy routes it to the right upstream endpoint.

Model catalog

Transparent routing (`provider/model`)

On the OpenAI surfaces (/v1/chat/completions and /v1/responses) the model field is parsed as provider/model:

You can also force the provider with ?p=<provider> and send a bare model id.

Adding an OpenAI-compatible upstream provider

forge and nebius are both configurable providers: any OpenAI-compatible upstream wired up from exactly two env vars, <PROVIDER>_BASE_URL (scheme, host, port, path prefix) and <PROVIDER>_API_KEY (sent as Authorization: Bearer <key>) — no code change needed to point either one at a different host. They differ in one way: whether the base URL has a working default.

Both http:// and https:// are supported, with any host/port/path prefix — unlike every fixed-hostname provider above, these two read their wire format from the env var rather than a hardcoded https:// + well-known host. An unset FORGE_BASE_URL (no default) or a malformed override on either provider makes forge/.../nebius/... requests fail with a clear 4xx, never a crash.

Both work on all three surfaces (/v1/messages, /v1/chat/completions, /v1/responses) with the same Anthropic↔OpenAI translation, streaming, and tool-cap handling as every other OpenAI-compatible provider. GET /v1/models additionally proxies GET ${FORGE_BASE_URL}/models and merges the results into the default pack's listing (ids prefixed forge/) — forge-only, since its LoRA adapters are registered on the server itself rather than known ahead of time; nebius's catalog is the well-known set of ids you already pass in.

# forge — self-hosted vLLM, no auth
curl http://localhost:18090/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"forge/my-lora-v3","messages":[{"role":"user","content":"hi"}]}'

# nebius — hosted, requires NEBIUS_API_KEY
curl http://localhost:18090/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"nebius/meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"hi"}]}'

Named endpoints — N local/LAN model servers (`~/.agentproto/llm-endpoints.json`)

forge is one keyless self-hosted server. Real setups often have several — Ollama, llama-server, vLLM — each on its own host/port, possibly on another LAN machine. Named endpoints generalize forge's mechanism to N of them, configured from a JSON file instead of a pair of env vars per server:

// ~/.agentproto/llm-endpoints.json (path overridable via LLM_ENDPOINT_ENDPOINTS_FILE)
{
  "endpoints": [
    {
      "id": "bonsai",
      "kind": "openai",
      "baseUrl": "http://192.168.1.20:8081/v1",
      "apiKeyEnv": "BONSAI_API_KEY",
      "defaultRequestFields": { "chat_template_kwargs": { "enable_thinking": false } },
      "timeoutMs": { "firstTokenMs": 180000 }
    },
    { "id": "ollama", "kind": "openai", "baseUrl": "http://192.168.1.20:11434/v1", "connector": "ollama" },
    {
      "id": "lmstudio",
      "kind": "openai",
      "baseUrl": "http://127.0.0.1:1234/v1",
      "connector": "lmstudio",
      "defaultRequestFields": { "reasoning_effort": "none" }
    }
  ]
}

The field names deliberately mirror openagentik/router's providers[] schema (kind, baseUrl, apiKeyEnv, defaultRequestFields, timeoutMs) so a config is portable between the two. Each entry becomes a routable provider <id>/<model> — bonsai/bonsai-27b, ollama/qwen2.5-coder — going through the exact same dispatch path as forge/nebius above: all three surfaces, Anthropic↔OpenAI translation, streaming, tool-cap handling, and GET /v1/models merging (an unreachable endpoint is skipped with a warning, never a 500 for the whole listing). forge itself keeps working unchanged — it's the implicit endpoint that FORGE_BASE_URL/FORGE_API_KEY configure; a file entry may not reuse the id "forge".

connector names which local/LAN runtime baseUrl points at — lmstudio, ollama, vllm, llama-server, or the generic openai-compatible fallback (see src/connectors.ts). Optional: it never affects request routing, only which connector's listModels/request quirks agentproto llm endpoints test/detect/doctor use to show loaded models and context size. An entry with no connector is treated as openai-compatible.
apiKeyEnv names an env var (never the key itself). Absent, or the env var unset, means the endpoint is always keyless — no Authorization header sent, same as forge with no FORGE_API_KEY.
defaultRequestFields is any set of top-level OpenAI-compatible request fields, merged UNDER the client's own request (a client-supplied top-level key always wins; for an object-valued key present on both sides — e.g. chat_template_kwargs — the merge goes one level deep, client sub-keys winning). It cannot set model, messages, stream, tools, or input — those would override routing/auth, not just default a parameter.
- chat_template_kwargs is the vLLM OpenAI-compatible extension forge already documents above. Needed in practice: Qwen3.6-based models (e.g. a Bonsai-served 27B) default to a "thinking" chat template and, on a tight max_tokens budget, can spend the whole budget reasoning and return empty content — enable_thinking: false avoids that unless the caller explicitly opts back in.
- A plain top-level field works the same way for a server that doesn't honour chat_template_kwargs or Anthropic's thinking/OpenAI's reasoning params at all: LM Studio serving prism-ml/bonsai-27b ignores both and only respects a top-level reasoning_effort: "none" to cut reasoning to 0 tokens. On the Anthropic /v1/messages surface, an explicit client thinking: {type: "enabled"} withholds a default reasoning_effort rather than forcing it off.
When the upstream still returns only reasoning_content (no visible content) — a model-level defect, not a config one — set LLM_ENDPOINT_PASSTHROUGH_THINKING=1 to surface it as an Anthropic thinking block instead of an empty message.
GET /v1/endpoints (and /endpoints) reports live health per endpoint — forge (if configured) plus every named endpoint, never nebius (a hosted provider with a well-known catalog, not a local/LAN server): {id, baseUrl (no credential), reachable, models, latencyMs}. Gated by the same access-token check as /v1/upstreams (not the /v1/models public exemption).
A packs.local.json route may target a named endpoint id as its provider — an id that resolves to neither a canonical upstream, forge, nebius, nor a configured endpoint is a clear load-time error instead of a confusing 400 the first time a client hits that code.

curl http://localhost:18090/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"bonsai/bonsai-27b","messages":[{"role":"user","content":"hi"}]}'

curl http://localhost:18090/v1/endpoints

Anthropic Messages surface (`/v1/messages`)

The default public pack lists provider-transparent model IDs (the same values you would send to the upstream provider). Real Claude model IDs are preserved only when they route to an actual Anthropic target.

If you need the claude CLI or another Anthropic-only client to accept non-Anthropic backends, either flip on the Anthropic-style format (below — works with any pack) or create a local compatibility pack (further below). Both surface equivalentClaudeName aliases that are matched only when enabled.

`coding` pack (curated OpenRouter coding models)

The committed coding pack is a small, production-only portfolio of coding models, all routed through OpenRouter's native Anthropic-compatible endpoint:

Select it like any pack (X-Proxy-Pack: coding, ?pack=coding, or /v1/coding/messages). The list is curated by hand against OpenRouter's live Models API/rankings (GET https://openrouter.ai/api/v1/models?supported_parameters=tools); availability and pricing drift, so re-check before relying on a route. Anything outside the pack is still reachable via transparent openrouter/vendor/model routing.

Anthropic-style format (`?format=anthropic`)

Any pack can be relabeled on the fly so an Anthropic-only client gets Claude-shaped model ids — without impersonating a real Anthropic model. Send X-Proxy-Format: anthropic (or ?format=anthropic) and each route's id becomes an opaque, deterministic claude-<family>-<sha> value, where the family comes from the route's tier (extra-high→fable, high→opus, medium→sonnet, small→haiku) and the suffix is a sha of the upstream id. The real route stays as the model's display_name.

Discover the current ids, then use one as the model:

# Discover (Anthropic-formatted model list)
curl -s -H 'anthropic-version: 2023-06-01' -H 'X-Proxy-Format: anthropic' \
  http://localhost:18090/v1/coding/models
# → { "data": [ { "id": "claude-opus-5246108", "display_name": "anthropic/claude-opus-4.8", … }, … ] }

# Use it (Messages path resolves the id back to the real OpenRouter route)
env -u ANTHROPIC_API_KEY \
  ANTHROPIC_BASE_URL="http://localhost:18090" \
  ANTHROPIC_CUSTOM_HEADERS="X-Proxy-Pack: coding, X-Proxy-Format: anthropic" \
  ANTHROPIC_AUTH_TOKEN="unused-the-proxy-holds-the-real-key" \
  ANTHROPIC_MODEL="claude-opus-5246108" \
  claude -p "…"

The ids are stable across restarts (they are derived from the upstream id, not random), so a discovered id keeps working until the underlying route changes.

Local compatibility packs

Create a packs.local.json file (gitignored) in the workspace root, the package root, or next to src/index.ts:

{
  "packs": {
    "local-claude": {
      "id": "local-claude",
      "label": "Local Claude compat",
      "description": "Claude-shaped aliases for my preferred backends",
      "models": {
        "my-opus": {
          "provider": "moonshot",
          "model": "kimi-k2.7-code",
          "equivalentClaudeName": "claude-opus-4-8"
        }
      }
    }
  }
}

Then select the pack via header (X-Proxy-Pack: local-claude), query param (?pack=local-claude), or URL path (/v1/local-claude/messages). The alias claude-opus-4-8 will route to moonshot/kimi-k2.7-code only on the Messages path and only when local-claude is active.

Driving the `claude` CLI through a local pack

Use the header, not the URL path. The claude binary appends /v1/messages to ANTHROPIC_BASE_URL itself, so a base of …/v1/local-claude becomes /v1/local-claude/v1/messages, which matches no pack route — the request silently falls back to the default pack and 400s with "Unable to resolve model". Point the base at the proxy root and select the pack by header:

env -u ANTHROPIC_API_KEY \
  ANTHROPIC_BASE_URL="http://localhost:18090" \
  ANTHROPIC_CUSTOM_HEADERS="X-Proxy-Pack: local-claude" \
  ANTHROPIC_AUTH_TOKEN="unused-the-proxy-holds-the-real-key" \
  ANTHROPIC_MODEL="claude-opus-4-8" \
  ANTHROPIC_SMALL_FAST_MODEL="claude-haiku-4-5" \
  claude -p "…"

Pin ANTHROPIC_SMALL_FAST_MODEL too, or the harness's background calls request a Claude tier the pack does not alias. Give reasoning models real max_tokens headroom: a thinking model can spend a small budget entirely inside its thinking block, and since those blocks are stripped (see below) the client then sees an empty content with stop_reason: max_tokens.

Per-request overrides (query string)

OpenAI Responses API facade (Codex custom providers)

The proxy exposes a focused POST /v1/responses endpoint that implements the OpenAI Responses API on top of the existing OpenAI-compatible chat/completions providers. Codex custom providers can set wire_api = "responses" and point their base URL at this proxy.

The facade is intentionally narrow: it supports the constructs that map cleanly to a chat/completions request and rejects everything else up front. It routes through the transparent provider/model surface, not through alias packs.

Supported

model — transparent provider/model reference (e.g. openai/gpt-4.1).
input — a plain string or an array of message items (input_text) and function_call_output items.
instructions — injected as a leading system message.
tools — only type: "function" tools are accepted.
tool_choice — "auto", "none", "required", or { type: "function", name }.
stream — when true, upstream SSE is re-emitted as Responses API SSE events (response.created, response.output_text.delta, response.completed, …).
Standard sampling params: max_output_tokens / max_tokens, temperature, top_p, parallel_tool_calls.
reasoning.effort — mapped to the upstream reasoning_effort parameter.

Explicitly unsupported (returns 400)

previous_response_id — the facade is stateless; each request is translated independently.
text.format / structured output.
Non-function tool types (e.g. web_search).
Image, audio, or other non-text content items.

Batches

POST /v1/messages/batches (and /v1/{pack}/messages/batches) exposes the Anthropic Message Batches API on top of the proxy's own routing — a client pointed at the proxy can use the Anthropic SDK's batches surface unchanged (client.messages.batches.create/retrieve/results/cancel/list/delete) with params.model being any model the active pack routes. Batch is a delivery mode, not a model: each item's params is the same Messages body the sync /v1/messages route understands, so per-item routing, tool-trimming, and translation are all reused, not reimplemented.

| Route | Behaviour | | :--- | :--- | | POST /v1/messages/batches | { requests: [{ custom_id, params }] }. Validated (unique custom_id, no stream/speed/fallbacks/forced tool_choice, resolvable model) — 400 lists every offending item. Returns a message_batch object. | | GET /v1/messages/batches/{id} | Aggregate status across sub-batches: processing_status, request_counts, results_url (once ended). | | GET /v1/messages/batches/{id}/results | 404 until ended, then JSONL — one { custom_id, result } line per item, cached after the first fetch so a repeat GET doesn't re-hit the provider. | | POST /v1/messages/batches/{id}/cancel | Fans out to every sub-batch; a provider that doesn't support cancel (OpenRouter) is recorded, not fatal. | | GET /v1/messages/batches | List, newest first (?limit=). | | DELETE /v1/messages/batches/{id} | Only once ended (else 409); best-effort forwarded to Anthropic for native sub-batches. |

Native vs emulated

A batch whose items resolve to several providers is split into per-provider sub-batches and re-aggregated by custom_id:

anthropic and openrouter run on that provider's own async Batch API — 50% of token price, up to a 24h window.
Everything else (moonshot, requesty, zai, groq, xai, openai) has no batch API of its own, so items run through a local-queue emulation: the proxy submits each item to its own /v1/messages over loopback, so tool caps, thinking-strip, and empty-turn retry all still apply. Full price — there is no provider-side discount to draw from.

Credentials

Batches reuse the same per-provider credential resolution as /v1/messages. One exception: if the resolved anthropic credential is a subscription OAuth token (sk-ant-oat…) rather than an API key, batch creation fails closed with a 401 — the Anthropic Batches API only accepts API keys, so the proxy never attempts to send a subscription token to it.

Config

Batch records outlive the process — on restart, a native sub-batch simply re-polls the provider by its stored id, and an unfinished local-queue sub-batch resumes only the items still missing a result.

Tool handling

Automatic trimming

The claude CLI loads its full MCP config (~/.claude + .mcp.json + skills), which often exceeds a provider's tool limit (e.g. Groq: 128 max → 400 'tools': maximum number of items is 128). The proxy truncates payload.tools to the provider cap (PROVIDER_MAX_TOOLS in src/index.ts, currently groq: 128) before reshaping tools for the provider. Providers without a cap (moonshot, openrouter, zai, openai) are untouched. Use ?tools= / ?notools=1 for finer control.

Orphaned tool calls

When tools is truncated, conversation history can still contain tool_use blocks for undeclared tools (e.g. the CLI's Agent sub-agent). Groq validates strictly and rejects: 400 tool call validation failed: attempted to call tool 'X' which was not in request.tools. During Anthropic→OpenAI conversion the proxy converts those orphaned tool_use (and their matching tool_result) into plain text ([Used tool X with args …] / [Tool result: …]), preserving context without breaking validation. Declared-tool tool_use blocks pass through normally as OpenAI tool_calls.

Running it

No bespoke start script — it's a normal workspace package, driven by pnpm scripts:

# Dev: run the server with hot-reload (tsx watch)
pnpm --filter @agentproto/llm-endpoint serve

# Watch-rebuild the bundle (tsup --watch) — for consumers importing the package
pnpm --filter @agentproto/llm-endpoint dev

# Build the bundle (dist/index.mjs + dist/cli.mjs + types)
pnpm --filter @agentproto/llm-endpoint build

# Run the built server
pnpm --filter @agentproto/llm-endpoint start

# Type-check
pnpm --filter @agentproto/llm-endpoint check-types

Override the port with LLM_ENDPOINT_PORT=18099 pnpm --filter @agentproto/llm-endpoint serve.

The package also exports start() and the underlying server for embedding:

import { start } from '@agentproto/llm-endpoint'
start(18090)

Live end-to-end suite

src/__tests__/e2e.live.ts hits a running proxy (localhost:18090) with real provider keys, so it is deliberately kept off the vitest glob (it is not a unit test). Start the server first, then:

pnpm --filter @agentproto/llm-endpoint test:e2e

Securing a public deployment

The proxy holds real upstream provider keys, so any host that can reach it can spend your credits. Never expose it publicly without an inbound gate.

Inbound access gate

Set LLM_ENDPOINT_ACCESS_TOKENS to a comma-separated allow-list of secret tokens. When it is set, every request must present a listed token as either Authorization: Bearer <token> or an X-Proxy-Access: <token> header — anything else gets 401. When the variable is unset the gate is open (no inbound auth): fine for localhost, unsafe for a public origin.

x-api-key is not accepted. Some Anthropic-compatible clients default to sending the credential as x-api-key; the gate only reads Authorization: Bearer and X-Proxy-Access. Set the client's auth scheme to Bearer.

LLM_ENDPOINT_ACCESS_TOKENS="$(openssl rand -hex 24)" \
  pnpm --filter @agentproto/llm-endpoint start

Public model discovery (optional)

Clients that auto-discover models (e.g. Claude Desktop's launch-time model fetch) probe GET /v1/models without a credential, so the access gate 401s them and the connection test fails. Set LLM_ENDPOINT_PUBLIC_MODELS=1 to exempt only the default model-list path (/v1/models, /models) from every gate. Pack-scoped lists (/v1/<pack>/models) and all other paths stay gated, so no pack config leaks. Unset (default) keeps discovery gated too.

Edge / WAF token layer

LLM_ENDPOINT_EDGE_TOKENS is a second, independent allow-list checked via the X-Edge-Auth: <token> header. It's meant to be enforced at the edge (a Cloudflare WAF rule in front of the tunnel) so unauthenticated traffic never reaches the origin — and it's also re-checked in-process as a fallback. Each layer is independent; unset means off. Reusing the same secret as LLM_ENDPOINT_ACCESS_TOKENS (via Authorization: Bearer) is fine too — one secret, both the edge rule and the app gate.

Generating the Cloudflare rule (`print-waf-rule`)

llm-endpoint print-waf-rule prints a Cloudflare custom-rule (wirefilter) expression that blocks any request lacking a valid token, so the secret lives in one place and the edge rule is generated, not hand-typed. It reads LLM_ENDPOINT_EDGE_TOKENS (→ X-Edge-Auth) when set, else LLM_ENDPOINT_ACCESS_TOKENS (→ Authorization: Bearer); --host <h> (or LLM_ENDPOINT_PUBLIC_HOST) scopes the rule to one hostname. OPTIONS preflight is always allowed.

LLM_ENDPOINT_ACCESS_TOKENS="$SECRET" llm-endpoint print-waf-rule --host llm.example.com
# → (http.host eq "llm.example.com" and http.request.method ne "OPTIONS"
#     and not any(http.request.headers["authorization"][*] eq "Bearer $SECRET"))

Paste the output into a Cloudflare Block custom rule. If you also enabled LLM_ENDPOINT_PUBLIC_MODELS, add a carve-out so discovery bypasses the edge as well: … and http.request.uri.path ne "/v1/models" and http.request.uri.path ne "/models" and ….

Exposing the port

Any HTTP tunnel or reverse proxy that forwards to http://localhost:18090 works (Cloudflare Tunnel, ngrok, a VPS + nginx, …). Whichever you pick: keep the access gate enabled, and consider an edge control as defense-in-depth (e.g. a Cloudflare WAF rule or Cloudflare Access policy keyed on the same token) so unauthenticated traffic is rejected before it ever reaches the origin.

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

@agentproto/llm-endpoint

Credentials this proxy handles

Quick start (client config)

Model catalog

Transparent routing (provider/model)

Adding an OpenAI-compatible upstream provider

Named endpoints — N local/LAN model servers (~/.agentproto/llm-endpoints.json)

Anthropic Messages surface (/v1/messages)

coding pack (curated OpenRouter coding models)

Anthropic-style format (?format=anthropic)

Local compatibility packs

Driving the claude CLI through a local pack

Per-request overrides (query string)

OpenAI Responses API facade (Codex custom providers)

Supported

Explicitly unsupported (returns 400)

Batches

Native vs emulated

Credentials

Config

Tool handling

Automatic trimming

Orphaned tool calls

Running it

Live end-to-end suite

Securing a public deployment

Inbound access gate

Public model discovery (optional)

Edge / WAF token layer

Generating the Cloudflare rule (print-waf-rule)

Exposing the port

Transparent routing (`provider/model`)

Named endpoints — N local/LAN model servers (`~/.agentproto/llm-endpoints.json`)

Anthropic Messages surface (`/v1/messages`)

`coding` pack (curated OpenRouter coding models)

Anthropic-style format (`?format=anthropic`)

Driving the `claude` CLI through a local pack

Generating the Cloudflare rule (`print-waf-rule`)