Routing through OpenRouter

OpenRouter is one of the providers HybridInference can route to, and it is the one a fresh checkout uses by default. This page covers the OpenRouter adapter specifically. For how routing works in general, see Architecture; for the model-registry syntax, see Adding a New Model.

The default catalog

With no environment variables and no deployment overlay, config resolution falls through to config/examples/models.openrouter.yaml, which registers three models against OpenRouter — two direct and one demonstrating a local-first hybrid route with OpenRouter as the fallback leg. Supplying one API key is enough to get a working gateway:

source .venv/bin/activate            # the env `make setup-dev` creates
export OPENROUTER_API_KEY=sk-or-...
uvicorn serving.servers.app:app --no-proxy-headers --host 127.0.0.1 --port 8080

Run it from the repository root: the default paths are relative, and the boot log confirms which registry was chosen with a line like Registered 3 routes from config/examples/models.openrouter.yaml.

curl -s --noproxy '*' http://127.0.0.1:8080/routing

The catalog is a starting point, not a fixture: OpenRouter’s model slugs move, and the pricing figures in that file are approximate and used only for the gateway’s own usage accounting. Copy it and edit freely.

Route syntax

An OpenRouter route is an entry in a model’s route: list. There are two forms of kind:.

`kind: openrouter`

OpenRouter picks the upstream provider itself, under its own default policy.

- kind: openrouter
  weight: 1.0
  base_url: https://openrouter.ai/api/v1
  api_keys:
    - ${OPENROUTER_API_KEY}
  provider_model_id: meta-llama/llama-3.3-70b-instruct

provider_model_id is OpenRouter’s own slug for the model. The id: your clients ask for is the gateway’s; the two are independent on purpose.

`kind: openrouter[<slug>]`

Pins every request on that leg to one OpenRouter upstream, by sending provider: {order: [<slug>], allow_fallbacks: false}.

- kind: openrouter[deepinfra]
  weight: 1.0
  base_url: https://openrouter.ai/api/v1
  api_keys:
    - ${OPENROUTER_API_KEY}
  provider_model_id: meta-llama/llama-3.3-70b-instruct

The slug must match [A-Za-z0-9_.-]+, optionally with /-separated segments (deepinfra, deepinfra/turbo). A malformed bracket form — empty pin, whitespace, nested or unmatched brackets — raises at registration rather than being silently ignored. OpenRouter’s list of provider slugs is at https://openrouter.ai/docs/features/provider-routing.

Two legs that share a base_url but pin different upstreams still get distinct endpoint_ids, because the bracketed kind survives into the identifier. Their circuit-breaker and availability state therefore stay isolated: one flaky upstream does not take the other out. In api_logs, though, both forms record provider = "openrouter", so analytics sees a single OpenRouter cohort.

Sort policy

An OpenRouter route created through the admin provider-routes API may carry an openrouter_sort policy of price, throughput, or latency, which is sent as provider: {sort: <policy>}. It applies only to unpinned routes — a route with a bracket-form pin sends order instead, and the pin wins. The API rejects the field with 422 on a non-OpenRouter route or an unrecognised value.

What the adapter sends

OpenRouterAdapter (apps/backend/serving/adapters/openrouter.py) is a thin subclass of the generic OpenAI-compatible adapter. On top of the normal request it adds:

Attribution headers. OpenRouter credits traffic — for its leaderboard and its free-tier limits — to the site these two headers name. X-Title carries your site’s name and HTTP-Referer its public address, taken from SITE_NAME and SITE_PUBLIC_BASE_URL or from an active distribution manifest. Without a public address no HTTP-Referer is sent, and without a site name X-Title is HybridInference. Set both if you want your own site credited.
usage: {include: true} on every request, so OpenRouter returns its per-request cost field.
stream_options: {include_usage: true} on streaming requests, so the final SSE chunk carries the usage block.
provider: {...} for the pinned and sort cases described above.

Cost accounting

Two numbers, deliberately kept apart:

Column in `api_logs`	Meaning
`cost_usd`	What the caller is charged: tokens × the pricing declared for the model in your registry. Unaffected by OpenRouter
`upstream_cost_usd`	What OpenRouter reported it charged you for that request

upstream_cost_usd is NULL for non-OpenRouter routes, and NULL for an OpenRouter route when the response carried no cost figure.

API keys

A route’s api_keys: is a list, and the adapter treats it as a pool even when it holds a single key. With several keys the pool rotates: a key that hits a key-specific or transient failure (429, 401/402/403, 408/425, 5xx, or a timeout) hands the request to the next key. The failing key is muted only once the request has nowhere left to rotate to, and then for 20 seconds. Request-scoped failures such as 400 and 422 fail identically on every key, so they propagate immediately instead of burning the pool.

An api_keys: entry that expands from an unset environment variable is treated as absent. A route marked optional: true is then skipped with a warning naming the model and kind; any other route raises MissingEnvBackedKeyError and startup fails. Either way a route is never registered pointing at an endpoint it cannot authenticate to.

Error handling

OpenRouter errors travel the same path as any other OpenAI-compatible provider’s. Two mechanisms handle them, and they are independent:

Fallback. When a request to an OpenRouter leg fails for any reason, the router records the failure and tries the model’s remaining legs in route order, skipping any that are admin-disabled, modality-incompatible, or circuit-open. The exception is a request the caller explicitly pinned with X-Route-Pin, which never falls back. If every leg fails, the first error is what the client sees. Failure status codes do not select between “fall back” and “propagate” — that decision is only about whether another leg is available.

The circuit breaker. Failures accumulate per endpoint_id; after CIRCUIT_FAILURE_THRESHOLD consecutive failures (default 3) the endpoint stops receiving traffic for CIRCUIT_COOLDOWN_SECONDS (default 30) before a half-open probe. Client errors are exempt: a 4xx other than 408, 429, 401, and 407 does not open the circuit, because letting one caller’s bad request take the endpoint away from everyone else is the cascade the exemption exists to prevent. 408 and 429 mean OpenRouter is overloaded and do count. 401 and 407 mean the gateway’s configured credential was rejected — 401 the OPENROUTER_API_KEY, 407 an egress proxy demanding its own — which no caller can work around, so they count and additionally page.

402 and 403 do not count against the circuit by themselves. On a route whose keys are listed under api_keys:, though, the gateway treats them as a problem with the key, as described under API keys: it moves on to the next key and, once none is left, sets the key aside for a while. An account with no credit, which answers 402 to everything, therefore soon has no usable key, and that does open the circuit. A route with a single api_key: has no pool, so there a 402 never opens it. Unlike 401 and 407, 403 does not page, because providers also use it to refuse single requests — content policy, region blocks — while the service itself is healthy.

Troubleshooting

Every OpenRouter request returns 401. The gateway’s key was rejected, not the caller’s. Check OPENROUTER_API_KEY in the environment the backend process actually sees. A key that is unset entirely never gets that far — the route is skipped or startup fails, as above — so a 401 means a key was present and OpenRouter refused it.

The model is missing from /v1/models. Confirm the registry the gateway loaded: the boot log emits Registered N routes from <path>, which is the file the resolution chain chose. If it is not the file you edited, an explicit MODELS_CONFIG_PATH or a distribution manifest is winning over it — see Configuration.

Requests reach the wrong upstream. Use GET /routing to see the effective weight distribution per model. Note that this endpoint is unauthenticated and discloses upstream base URLs; see the warning in Architecture.