@routerbench/edge-runner

Run an LLM on your own machine and plug it into RouterBench's smart router. Your model runs locally with node-llama-cpp, is exposed through a secure frp tunnel, and registers itself as a self-hosted provider in your org. When the smart router picks it, inference runs on your hardware — your data never leaves your machine except the response you return.

Install

# Global (recommended for regular use) — gives you an `edge-runner` command:
npm install -g @routerbench/edge-runner
edge-runner --model hf:Qwen/Qwen2.5-1.5B-Instruct-GGUF/qwen2.5-1.5b-instruct-q4_k_m.gguf --key rb_xxx

# Or run once with no install:
npx @routerbench/edge-runner --model ./models/qwen2.5-7b.gguf --key rb_xxx

--model takes a local .gguf path or a Hugging Face URI (hf:owner/repo/file.gguf). Leave it running; Ctrl-C cleanly deregisters.

The hf: filename must match the repo exactly (case-sensitive) or you get a 404. The example above is a small single file (~1 GB). Bigger single-file options: 3B hf:Qwen/Qwen2.5-3B-Instruct-GGUF/qwen2.5-3b-instruct-q4_k_m.gguf, 7B hf:bartowski/Qwen2.5-7B-Instruct-GGUF/Qwen2.5-7B-Instruct-Q4_K_M.gguf. The official Qwen/…-GGUF repos split large quants (7B+) into shards (…-q4_k_m-00001-of-00002.gguf — point at the first shard, rest auto-download).

You can also use env vars instead of flags: ROUTERBENCH_API_KEY, EDGE_MODEL, EDGE_GPU, EDGE_PRIORITY, EDGE_PORT, EDGE_CTX.

Run in the background (keep it always on)

A local model server should stay running. Use pm2 (auto-restarts on crash):

npm install -g pm2 @routerbench/edge-runner
pm2 start edge-runner --name my-model -- \
  --model hf:Qwen/Qwen2.5-1.5B-Instruct-GGUF/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  --key rb_xxx --gpu auto
pm2 save && pm2 startup       # (optional) start on boot
pm2 logs my-model

Or systemd (Linux) — a ~/.config/systemd/user/edge-runner.service running the global binary with Restart=always, then systemctl --user enable --now edge-runner. Or quick: nohup edge-runner … & / inside tmux. If it crashes, RouterBench drops it from routing within ~90s and your supervisor restarts it (which re-registers automatically).

Full docs: https://docs.routerbench.com/features/edge-runner

What it does

1. Loads the GGUF model via node-llama-cpp and serves an OpenAI-compatible POST /v1/chat/completions (streaming + non-streaming) on localhost.
2. Opens an frp tunnel to frp.routerbench.com, giving your server a public URL.
3. Registers with RouterBench (POST /edge/register) — creating a custom provider (baseUrl = tunnel, token = a per-run secret only the gateway gets) and a self-hosted router model with default capability scores.
4. Heartbeats every 30s; the model is dropped from routing if heartbeats stop.

Options

| Flag | Default | Description | |------|---------|-------------| | --model | — | Path to a .gguf or hf:owner/repo/file.gguf (required) | | --key | $ROUTERBENCH_API_KEY | Your RouterBench API key (required) | | --name | derived from model | Model id shown in the app / router pool | | --priority | high | How strongly the router prefers this model: high (you ran it to use it) | normal | low (fallback only) | | --gpu | auto | auto | off | number of layers | | --context-size | 8192 | Context window | | --port | 8770 | Local server port | | --gateway | https://api.routerbench.com | RouterBench API base |

Security

The tunnel URL is public but the model is not open: every request must present Authorization: Bearer <secret>, where <secret> is generated per run and handed only to the RouterBench gateway at registration. Anyone else hitting the URL gets 401.

Requirements

• Node.js ≥ 20
• Enough RAM/VRAM for your chosen model
• tar on PATH (used once to unpack the bundled frpc)