Local Models (Ollama, vLLM)
Route to your own Ollama, vLLM, LM Studio or TGI endpoint through RouterBench.
Local & Self-Hosted Models
Any OpenAI-compatible endpoint — Ollama, vLLM, LM Studio, TGI, or a fine-tuned model server — can be registered as a custom provider and routed to through RouterBench, with the same routing, guardrails, observability and structured-output support as hosted models.
1. Register your endpoint
Add a provider key (dashboard → Provider Keys → Custom) with:
| Field | Example (Ollama) |
|---|---|
name | ollama-local |
baseUrl | http://your-ollama-host:11434/v1 |
token | ollama (placeholder — Ollama ignores it) |
The RouterBench gateway must be able to reach the endpoint over the network. Run
RouterBench self-hosted / on-prem alongside Ollama, or expose the endpoint through
your VPC / connector. A cloud gateway cannot reach a localhost Ollama.
2. Route to it
curl https://api.routerbench.com/v1/chat/completions \
-H "Authorization: Bearer $ROUTERBENCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "model": "custom/ollama-local/llama3.1",
"messages": [{ "role": "user", "content": "Hello" }] }'import { RouterBench } from "@routerbench/ai-sdk-provider";
RouterBench("custom/ollama-local/llama3.1");Why route local models through RouterBench?
- One API for local and cloud models — switch with a model id.
- Policy routing — force sensitive prompts to the local model, burst the rest to cloud.
- Guardrails, observability, structured output work identically on local models.
- Fine-tuned / distilled models become routable candidates the router can pick.
Requirements: the endpoint must expose the OpenAI /v1/chat/completions format
(Ollama, vLLM, LM Studio and TGI all do).
How is this guide?