Edge Runner (run local models)

Run an LLM on your own machine with node-llama-cpp and let RouterBench's smart router route inference to it — cost $0, fully private.

@routerbench/edge-runner lets you run an LLM on your own hardware and plug it into RouterBench. Your model runs locally with node-llama-cpp, is exposed through a secure tunnel, and registers itself as a self-hosted model in your org. When the smart router picks it, inference runs on your machine — your data never leaves it except the response you return.

Cost $0 — it's your compute, nothing is billed per token. Private — inference runs locally; the tunnel is protected by a per-run secret only the gateway holds.

Requirements

Node.js ≥ 20
Enough RAM/VRAM for your chosen model
A RouterBench API key (Dashboard → API Keys, rb_...)

Install

Option A — global install (recommended for regular use)

npm install -g @routerbench/edge-runner
edge-runner --model hf:Qwen/Qwen2.5-1.5B-Instruct-GGUF/qwen2.5-1.5b-instruct-q4_k_m.gguf --key rb_your_key

Option B — no install (try it out)

npx @routerbench/edge-runner --model ./mymodel.gguf --key rb_your_key

--model accepts a local .gguf path or a Hugging Face URI (hf:owner/repo/file.gguf, downloaded automatically). On first run it also fetches the llama.cpp binary and the tunnel client.

The hf: filename must match a file that actually exists in the repo, exactly (case-sensitive) — a wrong name returns HTTP 404. The example above is a small single file (~1 GB) that downloads in a minute or two.

Bigger models (all verified single files):

3B (~1.9 GB): hf:Qwen/Qwen2.5-3B-Instruct-GGUF/qwen2.5-3b-instruct-q4_k_m.gguf
7B (~4.7 GB): hf:bartowski/Qwen2.5-7B-Instruct-GGUF/Qwen2.5-7B-Instruct-Q4_K_M.gguf

Note: the official Qwen/…-GGUF repos split large quants (7B+) into shards (e.g. qwen2.5-7b-instruct-q4_k_m-00001-of-00002.gguf) — point at the first shard and the rest download automatically, or use the single-file bartowski/… repo above. Browse a repo's Files tab on Hugging Face to copy an exact name.

Run

Normal (foreground)

edge-runner \
  --model hf:Qwen/Qwen2.5-1.5B-Instruct-GGUF/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  --key rb_your_key

You'll see ✓ Live — "<model>" is connected. Leave the terminal open; press Ctrl-C to stop (it deregisters cleanly).

With environment variables

Any flag can come from the environment instead:

export ROUTERBENCH_API_KEY=rb_your_key
export EDGE_MODEL=./mymodel.gguf
export EDGE_GPU=off        # optional
edge-runner                # picks up EDGE_MODEL + ROUTERBENCH_API_KEY

(EDGE_MODEL, ROUTERBENCH_API_KEY, EDGE_GPU, EDGE_PRIORITY, EDGE_PORT, EDGE_CTX.)

Run in the background (keep it always on)

A local model server should stay running. Pick one:

pm2 (survives terminal close, auto-restarts on crash)

npm install -g pm2 @routerbench/edge-runner
pm2 start edge-runner --name my-model -- \
  --model hf:Qwen/Qwen2.5-1.5B-Instruct-GGUF/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  --key rb_your_key --gpu auto
pm2 save            # persist the process list
pm2 startup         # (optional) start pm2 on boot — run the printed command
pm2 logs my-model   # follow output

If it ever crashes, RouterBench removes it from routing within ~90s and pm2 brings it back, which re-registers automatically.

systemd (Linux, start on boot)

~/.config/systemd/user/edge-runner.service:

[Unit]
Description=RouterBench edge-runner
After=network-online.target

[Service]
ExecStart=%h/.npm-global/bin/edge-runner --model %h/models/model.gguf --key rb_your_key --gpu auto
Restart=always
RestartSec=5

[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user enable --now edge-runner
journalctl --user -u edge-runner -f

Quick & dirty

nohup edge-runner --model ./model.gguf --key rb_your_key &
# or run it inside tmux / screen

Use your model

Once it's live, send an auto request with the same API key — the smart router routes it to your machine:

curl https://gateway.routerbench.com/v1/chat/completions \
  -H "Authorization: Bearer $ROUTERBENCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}'

The response comes back with "used_model": "<your model>" and "used_provider": "custom". Works from the CLI, desktop app, or any OpenAI-compatible client pointed at the gateway.

Options

Flag	Default	Description
`--model`	—	`.gguf` path or `hf:owner/repo/file.gguf` (required)
`--key`	`$ROUTERBENCH_API_KEY`	your RouterBench API key (required)
`--priority`	`high`	`high` = router prefers your model · `normal` · `low` = fallback only
`--gpu`	`auto`	`auto` · `off` · number of layers
`--context-size`	`8192`	context window
`--name`	derived from model	the model id shown in your dashboard
`--port`	`8770`	local server port
`--help` / `--version`		usage / version

Tool-calling requests are not routed to edge models (the local server doesn't implement function-calling) — those go to a hosted model automatically. Use --priority low if you want your model used only as an overflow/fallback behind hosted models.

How routing works

--priority high (default) scores your model above hosted models, so auto strongly prefers it.
The model appears in your org's smart-router pool; only your org can route to it.
If the runner stops, the model drops from routing automatically (liveness heartbeat) and returns once it's back.

Edge Runner (run local models)

On this page