Server & Backends¶

Server Section¶

Controls the HTTP server behavior:

server:
  bind_address: "0.0.0.0:8080"    # Host and port to bind
  workers: 4                       # Worker threads (0 = auto)
  connection_pool_size: 100        # HTTP connection pool size

Multiple Bind Addresses and Unix Sockets¶

The server supports binding to multiple addresses simultaneously, including Unix domain sockets (on Unix-like systems and Windows 10 1809+). This enables flexible deployment scenarios such as:

• Listening on both IPv4 and IPv6 addresses
• Exposing a TCP port for external clients while using a Unix socket for local services
• Running behind a reverse proxy via Unix socket for better security

Single Address:

server:
  bind_address: "0.0.0.0:8080"

Multiple Addresses:

server:
  bind_address:
    - "127.0.0.1:8080"           # IPv4 localhost
    - "[::1]:8080"               # IPv6 localhost
    - "0.0.0.0:9090"             # All interfaces on port 9090

Unix Socket Binding (Linux, macOS, and Windows 10 1809+):

server:
  bind_address:
    - "0.0.0.0:8080"             # TCP for external access
    - "unix:/var/run/continuum-router.sock"  # Unix socket for local services
  socket_mode: 0o660              # Optional: file permissions for Unix sockets (octal)

Configuration Options:

Unix Socket Notes:

• Unix socket addresses must start with unix: prefix
• Existing socket files are automatically removed before binding
• Socket files are cleaned up on graceful shutdown
• On Windows 10 1809+ (Build 17063+), Unix sockets are fully supported via the socket2 crate
• On other non-Unix platforms, unix: addresses log a warning and are skipped
• Windows does not support Unix file permission modes; the socket_mode option is accepted but ignored
• Unix socket connections bypass IP-based authentication checks (client IP reported as "unix")

Nginx Reverse Proxy Example:

upstream continuum {
    server unix:/var/run/continuum-router.sock;
}

server {
    listen 443 ssl;
    location /v1/ {
        proxy_pass http://continuum;
    }
}

Performance Tuning:

• workers: Set to 0 for auto-detection, or match CPU cores
• connection_pool_size: Increase for high-load scenarios (200-500)

CORS Configuration¶

CORS (Cross-Origin Resource Sharing) allows the router to accept requests from web browsers running on different origins. This is essential for embedding continuum-router in:

• Tauri apps: WebView using origins like tauri://localhost
• Electron apps: Custom protocols
• Separate web frontends: Development servers on different ports

server:
  bind_address: "0.0.0.0:8080"
  cors:
    enabled: true
    allow_origins:
      - "tauri://localhost"
      - "http://localhost:*"        # Wildcard port matching
      - "https://example.com"
    allow_methods:
      - "GET"
      - "POST"
      - "PUT"
      - "DELETE"
      - "OPTIONS"
      - "PATCH"
    allow_headers:
      - "Content-Type"
      - "Authorization"
      - "X-Request-ID"
      - "X-Trace-ID"
    expose_headers:
      - "X-Request-ID"
      - "X-Fallback-Used"
    allow_credentials: false
    max_age: 3600                   # Preflight cache duration in seconds

CORS Configuration Options:

Origin Pattern Matching:

Security Considerations:

• Using * for origins allows any website to make requests - only use for public APIs
• When allow_credentials is true, you cannot use * for origins - specify exact origins
• For development, use port wildcards like http://localhost:* for flexibility
• In production, always specify exact origins for security

Reload behavior: CORS is an Axum router layer constructed at startup. Restart after changing server.cors.

Backends Section¶

Defines the LLM backends to route requests to:

backends:
  - name: "unique-identifier"        # Must be unique across all backends
    backend_id: "stable-identity"    # Optional: stable accounting identity (see below)
    type: "generic"                  # Backend type (optional, defaults to "generic")
    url: "http://backend:port"       # Base URL for the backend
    weight: 1                        # Load balancing weight (1-100)
    api_key: "${API_KEY}"            # API key (optional, supports env var references)
    org_id: "${ORG_ID}"              # Organization ID (optional, for OpenAI)
    models: ["model1", "model2"]     # Optional: explicit model list
    internal: false                  # Optional: keep infrastructure models out of user surfaces (default false)
    enabled: true                    # Optional: set false to disable user traffic while staying probeable (default true)
    retry_override:                  # Optional: backend-specific retry settings
      max_attempts: 5
      initial_delay: "200ms"
      max_delay: "10s"
      backoff_multiplier: 2.0
      jitter: true
      retryable_status_codes: [429, 502, 503, 504]
      retryable_errors: [ConnectionError, TimeoutError]
      timeout: "30s"

Endpoint URL Rule¶

A backend url is a base URL, not a full request URL: the router appends each request path (/v1/chat/completions and the rest) to it, so anything after the path silently corrupts every request the backend receives. One endpoint rule therefore applies everywhere a backend URL enters the router, from config.yaml and the Admin backend API through hub-managed backend sync and the transient backend probe:

• The value must parse as an absolute URL.
• It must not embed credentials (user:password@). backends[].url is deliberately left readable by the configuration maskers, so a credential placed there is returned in the clear by GET /admin/config/full and rendered by the WebUI, while api_key is masked.
• It must not carry a query string or a fragment, not even an empty one (a bare trailing ? or #).
• The scheme must be a transport the router can dial: http://, https://, or unix://.

The rule is enforced identically everywhere as of this release: a violation is refused at configuration load, by startup, hot reload, the Admin backend API, hub config sync, and continuum-router config validate, which reports it as an error and exits non-zero. The message names the backend and the offending property and echoes no part of the URL, because the credentials case is exactly the one where echoing would leak a password into every surface that renders a configuration error. The same applies to the base rule on engine_stats.metrics_url; that field's additional constraints (http(s) scheme only, same host as the backend URL unless engine_stats.allow_external_metrics_url, never an https-to-http downgrade) were already hard errors and are unchanged.

These forms were deprecated in v1.26.0 with one release of notice and are rejected from v1.27.0.

Migrating a URL that embedded credentials¶

A backend URL with inline credentials did work: reqwest strips the userinfo and sends Authorization: Basic base64(username:password), which is why a self-hosted engine behind a credential-checking reverse proxy could be configured this way. Move the credential to api_key and name the scheme:

# Before, rejected from v1.27.0
- name: local-vllm
  type: vllm
  url: "http://svc:hunter2@10.0.0.5:8000/v1"

# After
- name: local-vllm
  type: vllm
  url: "http://10.0.0.5:8000/v1"
  api_key: "${VLLM_BASIC_CREDENTIAL}"   # "svc:hunter2"
  auth:
    type: basic

The bytes on the wire are unchanged, so the reverse proxy sees exactly what it saw before. What changes is that the credential now lives in a field the configuration maskers hide.

auth.type: basic is accepted on the self-hosted engine types and on continuumrouter: vllm, sglang, ollama, lmstudio, llamacpp, mlxcel, generic, and continuumrouter. api_key must hold username:password, split on the first colon per RFC 7617, so a password may itself contain colons. A cloud provider authenticates with its own scheme, so basic on one of those is a configuration error rather than a silently ignored field.

The configured scheme is enforced at every site that carries the credential, not only inside the shared request executor, and two guards keep it that way rather than a convention (#1529): the internal Bearer constructor is visible only inside the module that owns the scheme seam, so a new credential site outside it is a compile error, and a source audit (tests/authorization_scheme_audit_test.rs) fails the build on the remaining spellings unless the site is listed with a written reason for why the provider, rather than the operator, fixes its scheme.

type: continuumrouter joined the set in v1.27.0. It reaches a remote Continuum Router or Backend.AI GO instance, and that instance can sit behind the same credential-checking reverse proxy a local engine does, so the deployment shape auth.type: basic was added for applies to it unchanged. The scheme reaches every path that carries the credential, including the Unix socket model discovery path, which builds its Authorization header outside the shared request executor. Since v1.27.0 (#1528) that also covers the streaming chat-completions builders (the initial dispatch, every mid-stream fallback hop, auto backend selection, and the Unix socket streaming transport) and the /anthropic/v1/messages transform path that converts an Anthropic-API request onto an OpenAI-compatible engine, over HTTP and over a Unix socket alike; a mid-stream hop re-evaluates the scheme per hop, so a Bearer primary can hand off to a basic rescue backend. Since v1.27.0 (#1529) the remaining credential-carrying paths follow the configured scheme too: the /v1/responses conversion path for OpenAI-wire backends (ResponsesApiStrategy::ConvertToChatCompletions), non-streaming and streaming alike, /anthropic/v1/messages/count_tokens, the ACP session/prompt dispatch, the startup connection pre-warm, the hub-directed synthetic probe, the realtime WebSocket upgrade, and the streaming image generation, image variations, and image edit routes. The last two are included because their backend filter accepts any backend whose name or URL contains openai, so a generic backend named openai-proxy reaches them and generic accepts auth.type: basic; the filter itself is unchanged.

Since v1.27.0 (#1534) the registered OAuth strategy reaches the streaming builders too, not only the static credential. A backend that declares auth.type: oauth is dispatched on a streaming chat completion exactly as it is on the non-streaming path: the token is refreshed when it is at or near expiry, the strategy's Authorization is applied, and the provider-specific extra headers travel with it (originator and the Codex User-Agent, which the ChatGPT-subscription endpoint requires on every request, not just on the OAuth endpoints). That covers all four builders: the initial dispatch, every mid-stream fallback hop, auto backend selection, and the Unix socket streaming transport.

A mid-stream hop resolves the strategy per hop rather than reusing a token captured when the stream began. A rotation or a hot reload therefore takes effect on the following hop, and a hop that crosses providers carries its own backend's credential rather than the one the stream started with.

When a backend declares auth.type: oauth but no strategy loaded, which happens when its token store is missing or malformed, the request carries no Authorization at all. The static api_key never stands in for an absent token, matching the refusal the non-streaming path already made. One routing behavior is unchanged by this: a request naming a model that a ChatGPT-subscription backend serves is still translated onto /v1/responses, because that backend implements no /chat/completions endpoint, so the builders above serve the streaming routes that do not take that bridge.

Two schemes are honored only by the backend type that owns them: service_account by gemini, and sigv4 by bedrock. Declared on any other type, neither is applied anywhere, and the backend quietly authenticates with its static api_key as Bearer, or with no Authorization when it has no key. Since v1.27.0 both the startup log and continuum-router config validate warn about that combination, and a future release will reject it at load.

type: generic carries the credential too. Until v1.27.0 it did not: its factory arm can fall back to a backend built from a BackendInfo, which holds no credential, so a configured api_key was accepted and never sent, and the type worked against an authenticating endpoint only through the URL form this rule now refuses. Naming the specific engine type is still worth doing where one fits, because it also brings that engine's own health-check contract and its engine-statistics adapter, but it is no longer required in order to authenticate.

An unparseable URL remains a hard load error, as it always has been, and an empty url on a typed backend still means "use the type's default URL".

Stable Backend Identity¶

backend_id is an optional, operator-assigned identity for a backend. It matters only when the control plane is enabled: the router reports it to Continuum Hub in heartbeat inventory and stamps it on the usage records of the backend that actually served each request, which is what lets Hub charge usage to a per-backend cost center.

It exists because name cannot do the job. name is a display label an operator may rename at any hot reload, and a (provider, model) pair is not a backend, because several backends can serve one pair. Attributing cost to either would silently merge two distinct cost owners, or split one cost owner across a rename, and neither is visible after the usage has been billed.

For that reason the router never derives the value. It will not synthesize it from name, url, type, the served provider/model pair, list position, or a hash of any mutable field. You assign it, or the backend reports no identity at all.

backends:
  - name: "openai-primary"
    backend_id: "openai-eu-1"
    type: "openai"
    url: "https://api.openai.com/v1"

Rules¶

Stability¶

The value is stable across restarts because it is read from your configuration rather than minted at boot, and stable across renames because it is a separate field:

# Before: charged to cost owner "openai-eu-1"
backends:
  - name: "openai-primary"
    backend_id: "openai-eu-1"

---

# After a rename: still charged to "openai-eu-1"
backends:
  - name: "openai-frankfurt"
    backend_id: "openai-eu-1"

A hot reload that introduces a duplicate or malformed id is rejected as a whole, and the running router keeps serving on its last known good configuration rather than adopting a partly applied identity map.

What is reported¶

The id is resolved at the moment a backend is dispatched to, so a request that fell over to a second backend is attributed to the backend that actually produced the response, not to the one selection first picked. A provider batch carries the id captured when the job was submitted, so a completion hours later still names the backend that executed it. A response served from the router's local cache dispatched no backend and therefore reports no identity. The router never substitutes the display name when an id is absent.

Heartbeat inventory reports the same ids, which is how an operator discovers which ids exist to assign: Hub holds no backend inventory of its own.

Internal (Infrastructure-Only) Backends¶

Set internal: true on a backend to keep its models out of user-facing surfaces while still serving router-internal traffic (for example a guardrail classifier, an internal summarizer, or an embeddings model). The field defaults to false (public).

When a backend is internal:

• its models are excluded from GET /v1/models (and the /v1/models/extended, single-model /v1/models/{id}, and Anthropic /anthropic/v1/models listing variants);
• a user request routed to a model served only by internal backends is rejected with model-not-found;
• router-internal callers are unaffected: a guardrail provider that names the backend via its backend: reference resolves it by name, and admin surfaces still list it.

A model served by both an internal and a public backend stays visible and routes via the public one; the internal backend name is never leaked in listings.

backends:
  - name: "guard-classifier"
    type: "vllm"
    url: "http://localhost:8001"
    internal: true
    models: ["llama-guard-3"]

Disabled Backends¶

Set enabled: false on a backend to keep it configured but out of service. The field defaults to true, so omitting it leaves existing configurations unchanged.

When a backend is disabled:

• it is still constructed in the backend pool and stays health-checked and candidate-probeable, and it remains visible on admin and inventory surfaces;
• it stays discoverable on demand through the admin discovery endpoints (POST /admin/backends/{name}/models/discover and POST /admin/backends/probe), which query it live whenever an operator asks. It is not contacted by the periodic model-list aggregation behind GET /v1/models, since a backend that receives no traffic has nothing to contribute there and probing it every refresh only adds latency (a powered-off standby would otherwise spend the whole retry budget on connect timeouts, delaying the aggregation for every caller);
• it receives no request traffic at all, user or router-internal (unlike internal: true, which still serves internal callers);
• it contributes no models to the OpenAI or Anthropic model listings; a model served by both a disabled and an enabled backend stays visible and routes via the enabled one.

weight keeps its 1..=1000 range regardless of activation, so weight zero does not acquire a second meaning. Flipping enabled is hot-reloadable and takes effect without a restart.

backends:
  - name: "standby-vllm"
    type: "vllm"
    url: "http://localhost:8002"
    enabled: false
    models: ["llama-4-70b"]

Request Extensions¶

request_extensions lets an operator set a backend's routing preferences and attribution once, in configuration, instead of every client carrying them in every request. The motivating case is an OpenAI-compatible aggregator such as OpenRouter, which reads its routing instructions from the request body (provider, models, transforms) and its attribution from headers (HTTP-Referer, X-Title). It is a fixed JSON fragment merged into the body plus a fixed set of headers, both validated at configuration load; it is not a general request-rewriting facility, and there are no templates or expressions.

backends:
  - name: openrouter
    type: openai
    url: https://openrouter.ai/api/v1
    api_key: ${OPENROUTER_API_KEY}
    models: ["openai/gpt-5"]
    request_extensions:
      body:
        defaults:            # client wins
          provider:
            sort: throughput
        overrides:           # config wins
          provider:
            data_collection: deny
      headers:
        HTTP-Referer: https://backend.ai
        X-Title: Backend.AI GO

Merge Semantics¶

The body has two modes, mirroring request_params:

• defaults fills a key only when the client left it unset, meaning absent or JSON null. The client wins.
• overrides replaces whatever the client sent. The configuration wins. Use it for a policy a client must not be able to relax, such as provider.data_collection: deny.

Defaults apply first, then overrides. Both merge JSON objects deeply: when the client value and the configured value at the same key are both objects, the merge descends into them. Anything else is decided wholesale at that key: arrays are never merged element by element, and a type conflict (a client string where the configuration has an object) keeps the client value under defaults and takes the configured value under overrides.

For the example above, a client request carrying "provider": {"sort": "price", "data_collection": "allow"} reaches OpenRouter as:

{
  "provider": {"sort": "price", "data_collection": "deny"}
}

The client's sort survives because defaults never replaces a value the client set, and data_collection is replaced because overrides owns that one leaf. A request with no provider at all reaches the upstream with {"sort": "throughput", "data_collection": "deny"}.

Coverage¶

• Backend types. body is accepted only on backend types that send the OpenAI-compatible wire format: generic, openai, azure, vllm, sglang, tensorrt-llm, llamacpp, mlxcel, ollama, lmstudio, and continuumrouter. It is a load error on anthropic, gemini, and bedrock, whose native request bodies give a fragment such as provider no meaning. headers is accepted on every backend type except bedrock with endpoint_type: runtime or converse, where it is a load error: those requests are SigV4-signed inside the typed Bedrock backend, which has no seam for extra headers, so an accepted block would never be sent. Bedrock mantle, the default endpoint type, accepts it.
• Endpoints. The body fragment applies to /v1/chat/completions, /v1/completions, /v1/responses, and /v1/embeddings, including the converted ingresses that reach an OpenAI-wire backend (Anthropic Messages and the Responses API converted to Chat Completions). It never applies to model listing, images, audio, files, or rerank.
• Headers. Every request that carries the backend's own authentication carries the extension headers too, whether it serves a client or the router itself: inference on every ingress, model discovery (both the aggregated catalog and the auto-discovery a typed backend runs at construction when models is empty), health probes, connection pre-warm (the startup pass and the Anthropic and Gemini constructors'), backend type detection, llama.cpp /props capability detection, engine-statistics scrapes, Gemini context-cache management, provider batch lifecycle calls, synthetic control-plane probes, the guardrail backend: and smart-routing classifier calls through the typed backend executor, and the realtime WebSocket handshake. A configured header replaces a forwarded client header of the same name, and never an authentication header: those names are load errors. A backend with no key configured still gets them on its authenticated path. The block is read per send, so a hot reload, an Admin backend edit, or a control-plane config-sync change reaches the next call; an engine-statistics scrape picks it up when its poll task restarts, which a headers-only edit triggers. The Admin backend-create type probe runs before the candidate is validated, so it attaches the candidate's headers only when they already pass header validation, and otherwise probes without them and refuses the create. Health probes to a Unix-socket (unix:) backend send no extension headers, just as they send no credential. Health probe state is keyed by backend URL, so two backends that share a URL are probed with the header set of only one of them.
• Order. The fragment is merged last, into the fully built wire payload for the selected backend: after request-parameter policy, ingress conversion, and every provider-specific normalization.

Fallback and Retries¶

Extensions are applied per selected backend at dispatch, on every retry and every fallback hop, including pre-stream and mid-stream streaming fallback. A hop that moves to a different backend starts from the unextended request and carries that backend's extensions, or none, and never the previous backend's. Configure the block on every backend in a chain that needs it.

Validation¶

Every rule below is enforced at configuration load on every ingestion path (startup, hot reload, the Admin config and backend APIs, config import, control-plane sync, continuum-router config validate, and the MCP validate tool). Errors name the exact path, for example backends[2].request_extensions.body.overrides.temperature.

Reserved body keys, compared case-insensitively at the top level of defaults and overrides:

background (Responses) returns a queued response the client must poll instead of the result, and it needs store: true, which is already reserved. include and include_reasoning add output fields such as reasoning.encrypted_content and message.output_text.logprobs. prompt_cache_key is a per-caller cache key; one operator-wide value would put every caller in one cache bucket. safety_identifier is a per-end-user identity for OpenAI safety systems, the same class as user. modalities (Chat) with "audio" adds choices[].message.audio, whose voice and encoding audio (Chat) configures; a configured value would hand a client a response it did not ask for. prediction (Chat, Predicted Outputs) is content that comes from the client's own prompt, the same class as messages. guided_json, guided_regex, guided_choice, and guided_grammar are vLLM's legacy structured-output fields and structured_outputs is the unified field vLLM 0.12 replaced them with; each constrains generated output the same way the reserved response_format does. prompt_logprobs is vLLM's prompt-side counterpart to the reserved logprobs, adding log probabilities for the prompt tokens to the response. echo (legacy /v1/completions) prepends the prompt to the returned completion text, changing what the client's parser sees in choices[].text.

Sampling values belong in request_params, which types and range-checks them; this block is not a second path around it. stop, seed, and logit_bias are deliberately settable: they steer generation without changing the shape of the response a client parses, so an operator may pin them per backend, except where that backend's dispatch strips them (stop on a Codex backend). service_tier (a cost and latency tier on the operator's account; the response already reports the tier) and prompt_cache_retention / prompt_cache_options (cache retention and mode on the operator's account) stay settable for the same reason: they change nothing about identity, content, or response shape. Keys nested under an allowed top-level key are free-form, and null is refused anywhere in the fragment.

Reserved header names are compared case-insensitively and with every _ read as -, because several proxies and application servers fold the two (CGI-style environments map X-Api-Key and X_Api_Key to the same variable). x_api_key is therefore refused like x-api-key, and X_Title next to X-Title is a duplicate. The reserved names: Authorization, Proxy-Authorization, Host, Content-Length, Content-Type, Content-Encoding, Transfer-Encoding, Connection, Keep-Alive, Proxy-Connection, Upgrade, TE, Trailer, Expect, Cookie, Set-Cookie, Accept, Accept-Encoding, User-Agent, x-api-key, api-key, x-goog-api-key, x-auth-token, x-access-token, anthropic-version, anthropic-beta, OpenAI-Organization, OpenAI-Project, chatgpt-account-id, originator, X-Request-ID, X-Trace-ID, X-Correlation-ID, traceparent, tracestate, baggage, any header name renamed under tracing.headers, and every name starting with x-amz-, x-amzn-, or sec-websocket-. The router and its authentication layer set these itself, so an extension can never collide with or replace them.

Bounds:

A header value outside SP-~ (café, Hangul, a non-breaking space) is refused at load rather than sent as obs-text: HeaderValue::from_str at dispatch accepts bytes 0x80-0xFF, and RFC 9110 treats that byte range as obsolete, so upstreams and intermediaries handle it inconsistently, some reject the request and others read it as Latin-1. Percent-encode non-ASCII text instead. The error message never contains the header value.

Secrets¶

Header values support ${ENV_VAR} interpolation exactly like api_key. Every value under headers is treated as a secret whatever the header is called, because an aggregator key can travel under any header name: values are masked on the Admin config and backend reads, in YAML/TOML/JSON export, in config history, and in the WebUI, while header names stay visible. Sending a masked document back through PUT /admin/config/backends or PUT /admin/backends/{name} keeps the stored value. config diff and the MCP tools redact the values, extension header values are never logged, and config show --resolved prints them in plaintext exactly as it prints api_key. Body fragment values are not secrets and are shown as configured.

Hot Reload and Continuum Hub¶

A change to request_extensions applies to the next request without a restart: dispatch reads the backend's block from the published configuration, and the same publish refreshes the health checker's probe headers and revalidates the model list. The response cache follows the body fragment too: a backend with a non-empty body keeps its cache entries in a namespace derived from a fingerprint of the fragment, so a reload that edits defaults or overrides misses instead of serving answers produced under the previous fragment. Headers do not enter the cache key, and a backend without a body fragment keeps exactly the cache keys it had before. Separately from the response cache, the retry layer's request de-duplication answers a byte-identical non-streaming request that repeats within retry.timeout of an earlier one with that earlier result, before any backend is selected. It only coalesces requests served by the same published configuration, so any reload starts a fresh window and a repeat that follows one is dispatched on its own instead of replaying the previous fragment's answer. In AppProxy worker and ROUTER modes that window is shorter than retry.timeout in practice: each reconcile tick republishes the configuration whether or not anything changed, so the effective window is appproxy.reconcile_interval (default 15s).

The Continuum Hub v1 backend profile cannot carry the block. Hub-owned backends never have one, a hub sync never removes the block from a local backend that stays in the effective set, and a local backend export lists request_extensions as an unsupported field (never its values), which marks the entry as preview-only: the Hub must not adopt it. That is a Hub-side contract, not a Router-side refusal. In authoritative backend sync mode the Hub snapshot governs every backend not listed in immutable_backend_ids, and a local backend the snapshot replaces or removes loses its block with it, so list a backend whose extensions must survive under immutable_backend_ids.

Starting Without Backends¶

The router can start with an empty backends list (backends: []), which is useful for:

• Infrastructure bootstrapping: Start the router first, then add backends dynamically via the Admin API
• Container orchestration: Router container can be ready before backend services
• Development workflows: Test admin endpoints before backends are provisioned
• Gradual rollout: Start with zero backends and add them progressively

When running with no backends:

• /v1/models returns {"object": "list", "data": []}
• /v1/chat/completions and other routing endpoints return 503 "No backends available"
• /health returns healthy status (the router itself is operational)
• Backends can be added via POST /admin/backends

Example minimal configuration for dynamic backend management:

server:
  bind_address: "0.0.0.0:8080"

backends: []  # Start with no backends - add via Admin API later

health_checks:
  interval: "30s"
  timeout: "5s"
  unhealthy_threshold: 3
  healthy_threshold: 2
  endpoint: "/health"

logging:
  level: info
  format: json

admin:
  auth:
    method: bearer
    bearer_token: "${ADMIN_TOKEN}"

Backend Types Supported:

Native OpenAI Backend¶

When using type: openai, the router provides: - Default URL: https://api.openai.com/v1 (can be overridden for proxies) - Built-in model metadata: Automatic pricing, context windows, and capabilities - Environment variable support: Automatically loads from CONTINUUM_OPENAI_API_KEY and CONTINUUM_OPENAI_ORG_ID

Minimal OpenAI configuration:

backends:
  - name: "openai"
    type: openai
    models:
      - gpt-5.6-sol
      - gpt-5.6-terra
      - o3-mini

Full OpenAI configuration with explicit API key:

backends:
  - name: "openai-primary"
    type: openai
    api_key: "${CONTINUUM_OPENAI_API_KEY}"
    org_id: "${CONTINUUM_OPENAI_ORG_ID}"     # Optional
    models:
      - gpt-5.6-sol
      - gpt-5.6-terra
      - o1
      - o1-mini
      - o3-mini
      - text-embedding-3-large

Using OpenAI with a proxy:

backends:
  - name: "openai-proxy"
    type: openai
    url: "https://my-proxy.example.com/v1"   # Override default URL
    api_key: "${PROXY_API_KEY}"
    models:
      - gpt-5.6-sol

ChatGPT subscription / Codex headless login (OAuth device flow)¶

Continuum Router can authenticate against the OpenAI Codex backend (https://chatgpt.com/backend-api/codex) using a ChatGPT Plus / Pro / Enterprise subscription rather than a paid OpenAI API key.

The OpenAI provider does not implement standards-compliant RFC 8628 device flow; instead, the router uses OpenAI's custom Codex headless device-code flow, which is what the official Codex CLI uses for "login on headless devices." The flow has three steps:

1. Request a one-time user_code from auth.openai.com/api/accounts/deviceauth/usercode.
2. Poll auth.openai.com/api/accounts/deviceauth/token until the user approves the code in their browser.
3. Exchange the resulting authorization code for access / refresh tokens via PKCE at auth.openai.com/oauth/token.

Every request carries an originator: codex_cli_rs header so the Cloudflare front in front of auth.openai.com admits the traffic.

One-time login¶

Run the device-flow login from any machine that can open the OpenAI verification URL in a browser. The router prints a verification URL and a short user code, and polls the token endpoint until the device is approved.

continuum-router auth login --backend openai-chatgpt

Tokens are written atomically to the configured token_store with mode 0600 on Unix. After login, the router uses these tokens automatically and refreshes them transparently before they expire (60-second clock-skew margin). A 401 response from the backend triggers one forced refresh and a single retry before the error surfaces to the client.

Backend configuration¶

backends:
  - name: openai-chatgpt
    type: openai
    url: https://chatgpt.com/backend-api/codex
    auth:
      type: oauth
      oauth:
        provider: openai
        token_store: ~/.continuum-router/auth/openai.json
    # Codex backends enumerate their real models from the live endpoint (see
    # "Dynamic model enumeration" below). A non-empty list selects which
    # enumerated models to expose and is the fallback when enumeration fails or
    # returns nothing; leave it empty to expose every enumerated model.
    models:
      - gpt-5
      - codex-mini

client_id and scope default to the public Codex CLI values that auth.openai.com accepts; you only need to override them for a custom OAuth client registration.

Configuration reference¶

Dynamic model enumeration¶

Unlike static OpenAI API backends, the ChatGPT Codex backend does not expose a standard /v1/models endpoint. Instead it serves a model list at GET <base>/models?client_version=<ver> that is gated by the account's subscription plan. The router queries this endpoint with the loaded OAuth token during model discovery (and on the normal refresh cadence) and uses the result to populate /v1/models and routing:

• The request carries Authorization: Bearer <access_token>, the originator: codex_cli_rs header, and a chatgpt-account-id header derived from the id_token.
• Only user-facing models survive: an entry is kept when its visibility is list and its available_in_plans either is empty or contains the account plan (read from the id_token's chatgpt_plan_type claim). Internal entries such as codex-auto-review (visibility hide) are never exposed.
• The configured models: selection is applied to the enumerated set, exactly like every other backend. A non-empty list intersects the enumerated models down to the operator-selected subset; an empty list exposes the full enumerated set (e.g. gpt-5.5, gpt-5.4, gpt-5.4-mini). Enumerated models carry the clean owner openai instead of the raw backend name.
• On any failure (network error, non-2xx, empty list), the router falls back to the configured models: list so routing degrades gracefully. Dynamic enumeration applies only to Codex OAuth backends; other OAuth and static-key backends use their configured model lists.

Request handling¶

The ChatGPT Codex backend serves a single inference endpoint, <base>/responses (https://chatgpt.com/backend-api/codex/responses for the default base), which implements a stricter subset of the public OpenAI Responses API. The router converts /v1/chat/completions requests routed to a Codex OAuth backend into Responses-API form and adjusts the converted request to the shape Codex accepts.

• input is always sent as an item list; Codex rejects the bare-string shorthand that a single-message request would otherwise produce.
• max_output_tokens, temperature, top_p, presence_penalty, frequency_penalty, and stop are removed, because Codex rejects them.
• A tool_choice that forces a specific function is downgraded to "required", with a warning in the log. Codex supports only the string modes, so the model is still pushed to call a tool without naming one.
• When the request carries no system message, the instructions field is filled with a minimal default, because Codex rejects requests without instructions.

Codex accepts exactly one streaming/storage combination, stream: true with store: false, and the router forces both on every converted upstream call. A streaming client receives the converted SSE stream as usual. For a non-streaming client (stream: false), the router consumes the upstream SSE stream and folds it into a single chat-completion JSON body before responding; this detection also tolerates Codex responses that carry an SSE body without the text/event-stream content type. Because store is always false, nothing is persisted on the OpenAI side, and conversation state travels in the request as with any Chat Completions client.

Operational notes¶

• Tokens never appear in logs, traces, or metrics; only a short redacted prefix is logged when a refresh occurs.
• Refreshes are single-flighted with a tokio::sync::Mutex, so concurrent in-flight requests during expiry windows do not stampede the OAuth provider.
• Static api_key configurations are unaffected; OAuth is opt-in per backend via the auth.type: oauth block.
• Re-running continuum-router auth login --backend <name> rewrites the token store atomically and is safe while the router is running.
• The token store is read leniently: expires_at accepts epoch seconds (the canonical form the router writes), a numeric string, or an RFC3339 datetime, so a token file produced by another tool loads without editing.
• The OpenAI provider's auth endpoint is Cloudflare-protected; the default user_agent and originator: codex_cli_rs header mirror the official Codex CLI so the device flow reaches the OAuth endpoint instead of the bot-challenge page. Set auth.oauth.user_agent and/or auth.oauth.originator to custom values only if your environment specifically requires them.
• OAuth is rendered as oauth in YAML; o_auth is also accepted as an alias.

Environment Variables for OpenAI¶

Model discovery¶

The aggregated model service refreshes live backend catalogs and caches the result. For ordinary HTTP backends it requests the OpenAI-compatible model-list path derived from the backend URL; Unix-socket backends use /v1/models.

• A non-empty backends[].models list is an allowlist applied to the discovered result.
• Native Anthropic and Amazon Bedrock have no model-list endpoint. Configure models explicitly; validation requires it for operational Anthropic and Bedrock backends. The aggregation serves these types straight from the configured models list and makes no HTTP request for them, so a Bedrock backend never spends its retry budget on a /v1/models that does not exist.
• Backends with enabled: false are skipped entirely: the aggregation makes no request to them and they contribute no models. Use the admin discovery endpoints when you need a disabled backend's live catalog.
• A 404 or 405 from a model-list endpoint is treated as permanent, like 401/403: the refresh fails that backend immediately instead of retrying a provider that plainly serves no model list. 5xx responses and connection errors still use the full retry budget.
• OAuth backends other than the ChatGPT Codex flow use their configured models list. A Codex-flow backend enumerates its account-visible models from its authenticated Codex model endpoint and then applies the configured allowlist when non-empty.
• A failed backend is omitted from that refresh. If every backend fails, the refresh returns and caches an empty model list. There is no per-provider hardcoded fallback catalog for ordinary discovery failures.
• model_configs supplies backend metadata and matching information; it is not a replacement for a required explicit models list.

Use POST /v1/models/refresh or POST /admin/models/refresh for an on-demand refresh. The public refresh route is rate-limited.

Use POST /admin/backends/{name}/models/discover when an admin client needs the live catalog from one exact backend before the configured models: allowlist is applied. This endpoint is read-only: it does not change models:, model aliases, health state, circuit-breaker state, or the aggregated model cache. Codex OAuth backends use the same account- and plan-aware discovery path as normal aggregation, while unsupported backend types return a machine-readable model_discovery_unsupported error instead of falling back to configured model names.

Use POST /admin/backends/probe before registration when a management client needs to validate a candidate URL/key and populate an initial model picker. The candidate uses the same backend type, URL composition, auth headers, parser, normalization, response-size cap, and model-count limit as registered discovery, but stays request-local and is not written to configuration, history, pools, health state, circuit state, caches, environment files, or token stores. Configured fallbacks accept models or model_configs and remain subject to the per-backend model-count limit. The endpoint reports health.credential_status separately from catalog.source so an unauthenticated health success cannot be mistaken for credential validity; transient OAuth health is explicitly unknown because its lifecycle-managed authentication strategy is unavailable before registration. Support is machine-readable via /admin/capabilities as transient_backend_probe_v1.

Upstream vendor and metadata¶

An OpenAI-compatible aggregator such as OpenRouter lists hundreds of models the router has no model-metadata.yaml entry for, and its catalog is the only source of their vendor, context window, output cap, and price. Discovery reads those fields from each /v1/models entry as follows.

Vendor (owned_by), resolved per entry against the normalized model id:

1. A real upstream owned_by is kept unchanged.
2. When the upstream sends no owned_by at all (absent, null, or blank) and the id has a vendor/model namespace, the first path segment is the vendor, exactly as sent. OpenRouter sends no owned_by, so anthropic/claude-haiku-4.5 reports anthropic and openrouter/auto reports openrouter. A segment that is itself a placeholder value (such as vllm/...) is not used.
3. Otherwise a placeholder (system, organization_owner, an empty value on an id without a namespace, and so on) is replaced with the backend type's owner, as before: type: openai reports openai, and types without a canonical owner such as generic keep the value.

The rule reads only the id's first path segment, so a path-like id with no owned_by reports that segment even when it is not a vendor: hf.co/org/model reports hf.co. An explicit placeholder is not the same as a missing value: OpenAI's own catalog sends owned_by: "system", which still becomes openai. A per-model owned_by in model-metadata.yaml or backend model_configs still overrides the result, and response_defaults.owned_by still applies only to a value that remains a placeholder, so a namespace-derived vendor outranks that global default.

Context window and output cap: an entry's context length is read from max_model_len (vLLM, SGLang) or context_length (OpenRouter); when both are present, max_model_len wins. It fills limits.context_window when the effective metadata has no limits, within 1..=10000000. limits.max_output is the entry's top_provider.max_completion_tokens when that is present and between 1 and the same entry's context length, and the context length otherwise. When several backends serve one id, both values are the minimum across backends.

Pricing: pricing.prompt and pricing.completion are read as USD per token (a JSON string or number) and published as pricing.input_tokens and pricing.output_tokens in USD per 1M tokens, so "0.000001" becomes 1.0. Both must be valid or no price is published. A component that does not parse to a finite, non-negative number, including OpenRouter's "-1" for a variable price, means no price rather than zero, and so does a converted price above 100,000 USD per 1M tokens. pricing.input_cache_read becomes cached_input_discount (1 - cache_read / prompt) when that ratio is between 0 and 1. Other pricing keys, such as input_cache_write and web_search, are ignored, and a pricing value in any other shape maps nothing. Published prices feed smart-routing tier inference and cost scoring like configured ones.

Precedence for limits: configured metadata wins. The upstream context window and output cap fill limits only when the metadata matched from model-metadata.yaml, model-metadata.d/, backend model_configs, or the built-in OpenAI catalog has no limits.

Precedence for pricing: a price is specific to the provider serving the model, so an upstream price wins over pricing from the shared model-metadata.yaml (including model-metadata.d/ drop-ins) and the built-in OpenAI catalog, however that entry was matched: exact id, alias, date or quantization normalization, or vendor/ prefix stripping. Without this, a shipped entry priced 0/0 for self-hosting would make an aggregator's minimax/minimax-m2 read as free. Only a price set for that backend in backends[].model_configs wins over its upstream price. When at least one backend serving the id reports a usable upstream price, the published price is built from one candidate per backend: its model_configs price if it pins one, otherwise its upstream price; a backend with neither contributes nothing. The candidates combine as the maximum of each price component and the minimum cached_input_discount (a candidate without a discount clears it), so a pin replaces its own backend's upstream price but never lowers another backend's. The result is the highest price among the backends that state one; a backend whose price is missing or "-1" is left out, so the published price does not bound what that backend charges. When no backend reports a usable upstream price, pricing comes from configured metadata as before.

Every other metadata field, including owned_by, keeps configured metadata first. An upstream metadata object is never trusted as router metadata.

Where it appears: GET /v1/models keeps its OpenAI field set (id, object, created, owned_by); the only change there is the vendor. GET /v1/models/extended carries the mapped metadata and also passes every other upstream field through unchanged (pricing, context_length, architecture, supported_parameters, top_provider, canonical_slug, ...), except the keys the router writes itself (id, object, created, owned_by, backends, metadata, tier, domains), which are never duplicated. GET /v1/models/{model} carries the mapped metadata and max_tokens; a namespaced id's / works raw or percent-encoded (/v1/models/anthropic/claude-haiku-4.5 and /v1/models/anthropic%2Fclaude-haiku-4.5 are equivalent). The segments extended, proxy, and refresh are reserved for their own routes.

Native Gemini Backend¶

When using type: gemini, the router provides: - Default URL: https://generativelanguage.googleapis.com/v1beta/openai (OpenAI-compatible endpoint) - Built-in model metadata: Automatic context windows and capabilities for Gemini models - Environment variable support: Automatically loads from CONTINUUM_GEMINI_API_KEY - Extended streaming timeout: 300s timeout for thinking models (gemini-3.1-pro, gemini-3-flash, gemini-2.5-pro) - Automatic max_tokens adjustment: For thinking models, see below

Minimal Gemini configuration:

backends:
  - name: "gemini"
    type: gemini
    models:
      - gemini-3.1-pro-preview
      - gemini-3-flash-preview
      - gemini-2.5-pro
      - gemini-2.5-flash

Full Gemini configuration with API Key:

backends:
  - name: "gemini"
    type: gemini
    api_key: "${CONTINUUM_GEMINI_API_KEY}"
    weight: 2
    models:
      - gemini-3.1-pro-preview
      - gemini-3-flash-preview
      - gemini-2.5-pro
      - gemini-2.5-flash

Gemini Authentication Methods¶

The Gemini backend supports two authentication methods:

API Key Authentication (Default)¶

The simplest authentication method using a Google AI Studio API key:

backends:
  - name: "gemini"
    type: gemini
    api_key: "${CONTINUUM_GEMINI_API_KEY}"
    models:
      - gemini-3.1-pro-preview

Service Account Authentication¶

For enterprise environments and Google Cloud Platform (GCP) deployments, you can use Service Account authentication with automatic OAuth2 token management:

backends:
  - name: "gemini"
    type: gemini
    auth:
      type: service_account
      key_file: "/path/to/service-account.json"
    models:
      - gemini-3.1-pro-preview
      - gemini-3-flash-preview

Using environment variable for key file path:

backends:
  - name: "gemini"
    type: gemini
    auth:
      type: service_account
      key_file: "${GOOGLE_APPLICATION_CREDENTIALS}"
    models:
      - gemini-3.1-pro-preview

Service Account Authentication Features:

Creating a Service Account Key:

1. Go to Google Cloud Console
2. Navigate to IAM & Admin > Service Accounts
3. Create a new service account or select an existing one
4. Click Keys > Add Key > Create new key
5. Choose JSON format and download the key file
6. Store the key file securely and reference it in your configuration

Required Permissions:

The service account needs the following roles for Gemini API access:

• roles/aiplatform.user - For Vertex AI Gemini endpoints
• Or appropriate Google AI Studio permissions for generativelanguage.googleapis.com

Authentication Priority¶

When multiple authentication methods are configured:

If both api_key and auth are specified, the auth block takes precedence and a warning is logged.

Gemini Thinking Models: Automatic max_tokens Adjustment¶

Gemini "thinking" models (gemini-3.1-pro, gemini-3-flash, gemini-2.5-pro, and models with -pro-preview suffix) perform extended reasoning before generating responses. To prevent response truncation, the router automatically adjusts max_tokens:

This ensures thinking models can generate complete responses without truncation due to low default values from client libraries.

Environment Variables for Gemini¶

Native Anthropic Backend¶

When using type: anthropic, the router provides: - Default URL: https://api.anthropic.com (can be overridden for proxies) - Native API translation: Automatically converts OpenAI format requests to Anthropic Messages API format and vice versa - Anthropic-specific headers: Automatically adds x-api-key and anthropic-version headers - Environment variable support: Automatically loads from CONTINUUM_ANTHROPIC_API_KEY - Extended streaming timeout: 600s timeout for extended thinking models (Claude Opus, Sonnet 4)

Minimal Anthropic configuration:

backends:
  - name: "anthropic"
    type: anthropic
    models:
      - claude-sonnet-5
      - claude-haiku-4-5

Full Anthropic configuration:

backends:
  - name: "anthropic"
    type: anthropic
    api_key: "${CONTINUUM_ANTHROPIC_API_KEY}"
    weight: 2
    anthropic_fast_mode: false       # opt-in: enable fast mode for eligible models (default false)
    models:
      - claude-fable-5
      - claude-mythos-5   # limited release (Project Glasswing); needs approved access
      - claude-opus-4-8
      - claude-opus-4-7
      - claude-opus-4-6
      - claude-sonnet-4-6
      - claude-haiku-4-5

Anthropic API Translation¶

The router automatically handles the translation between OpenAI and Anthropic API formats:

When Anthropic returns stop_reason: "refusal", the router maps it to the OpenAI-compatible finish_reason: "content_filter". The upstream stop_details object (containing the refusal category) is forwarded on the response choice under stop_details and omitted when absent.

Example Request Translation:

OpenAI format (incoming from client):

{
  "model": "claude-sonnet-4-20250514",
  "messages": [
    {"role": "system", "content": "You are helpful."},
    {"role": "user", "content": "Hello"}
  ],
  "max_tokens": 1024
}

Anthropic format (sent to API):

{
  "model": "claude-sonnet-4-20250514",
  "system": "You are helpful.",
  "messages": [
    {"role": "user", "content": "Hello"}
  ],
  "max_tokens": 1024
}

Anthropic Native API Endpoints¶

In addition to routing OpenAI-format requests to Anthropic backends, the router also provides native Anthropic API endpoints:

These endpoints allow clients that use Anthropic's native API format (such as Claude Code) to connect directly without any request/response transformation overhead.

Claude Code Compatibility¶

The Anthropic Native API endpoints include full compatibility with Claude Code and other advanced Anthropic API clients:

Prompt Caching Support:

The router preserves cache_control fields throughout the request/response pipeline:

• System prompt text blocks
• User message content blocks (text, image, document)
• Tool definitions
• Tool use and tool result blocks

Header Forwarding:

Cache Usage Reporting:

Streaming responses from native Anthropic backends include cache usage information:

{
  "usage": {
    "input_tokens": 2159,
    "cache_creation_input_tokens": 2048,
    "cache_read_input_tokens": 0
  }
}

Anthropic's input_tokens excludes both cache buckets. When the same backend answers /v1/chat/completions, the router follows the OpenAI convention instead: prompt_tokens is input_tokens + cache_read_input_tokens + cache_creation_input_tokens, prompt_tokens_details.cached_tokens carries the cache reads, and the top-level cache_read_input_tokens and cache_creation_input_tokens are kept. Bedrock Converse is mapped the same way (inputTokens + cacheReadInputTokens + cacheWriteInputTokens). Before #1642 prompt_tokens on these backends left the cached prompt out, so a client that reads it sees a larger value on a cache hit.

The reverse bridge follows the Anthropic convention. When /anthropic/v1/messages is served by an OpenAI-compatible backend (Chat Completions) or a Responses API backend, the router reports input_tokens as the upstream prompt count minus the cache reads (prompt_tokens_details.cached_tokens, input_tokens_details.cached_tokens, or a top-level cache_read_input_tokens) and minus the cache writes, and returns those buckets as cache_read_input_tokens and cache_creation_input_tokens, in the non-streaming body and in the streaming message_start/message_delta usage alike. A usage block that is already Anthropic-shaped passes through unchanged. Before #1642 input_tokens on this bridge still contained the cached prompt, so a client that adds input_tokens + cache_read_input_tokens + cache_creation_input_tokens (a context-window calculator) counted the cached prompt twice; on a cache hit that client now sees a smaller input_tokens from the non-streaming body, and from a bridged stream reconstructed within this same router (for example its own local cache replay, or TokenUsage::from_json_value re-parsing the final message_delta values). A downstream router that meters a bridged stream with its own native Anthropic streaming tracker lands on the same total: message_start on a bridged stream carries input_tokens: 0 and the real input and cache values arrive only in message_delta, so the tracker reads input_tokens, cache_read_input_tokens, and cache_creation_input_tokens from message_delta as well (the Anthropic Messages API itself repeats those cumulative counts there) instead of recording a prompt count of 0 for that stream.

/v1/responses served by an Anthropic backend follows the Responses API convention: input_tokens is the inclusive prompt count and input_token_details carries cached_tokens (cache reads) and cache_write_tokens (cache writes), on the non-streaming body and on the stream's response.completed alike.

Tool-Call ID Normalization:

tool_use.id is scoped to the whole conversation in the Anthropic protocol, but some OpenAI-compatible backends behind this endpoint only guarantee an id unique within a single response (vLLM's Kimi-K3 template restarts its {tool_name}:{index} counter on every response). The router rewrites an outbound id that isn't already known to be conversation-unique to crt_<nonce:12><base64url_nopad(raw_id)> and restores the original bytes when the client replays it, so tool_use/tool_result pairing stays correct across turns with no configuration required. Ids already unique across the conversation (the toolu_, call_, fc_, and chatcmpl-tool- prefixes) pass through untouched.

Anthropic Extended Thinking Models¶

Models supporting extended thinking (Claude Opus, Sonnet 4, Claude Opus 4.7, Claude Opus 4.8, and Claude Sonnet 5) may require longer response times. The router automatically:

• Sets higher default max_tokens (16384) for thinking models
• Uses extended streaming timeout (600s) for these models

Claude Opus 4.7/4.8 and the Mythos-class models (Fable 5, Mythos 5) require the adaptive thinking API (thinking.type == "adaptive" + output_config.effort) and reject the legacy budget_tokens shape. The router normalizes explicit legacy thinking.type == "enabled" requests for these models to adaptive thinking. These models also do not accept temperature, top_p, or top_k; the router drops these parameters automatically. Fable 5 and Mythos 5 additionally reject an explicit thinking.type == "disabled" (HTTP 400); the router omits the thinking parameter entirely for claude-fable-5-* and claude-mythos-5-* in that case.

Claude Fable 5¶

Claude Fable 5 (claude-fable-5-*, alias claude-fable-5-latest) is Anthropic's most capable model, a Mythos-class flagship positioned a tier above Opus 4.8. Key characteristics:

• Context window: 1M tokens (input)
• Max output: 128K tokens
• Pricing: $10 / $50 per million tokens (input / output)
• Thinking: Adaptive only (thinking.type: "adaptive" + output_config.effort). The legacy enabled + budget_tokens shape returns HTTP 400, and an explicit thinking.type: "disabled" also returns HTTP 400 (the router omits the thinking parameter instead).
• Effort: Supports low, medium, high, and max (reasoning_effort: "xhigh" maps to max).
• Sampling parameters: temperature, top_p, top_k are not accepted. The router drops them automatically before forwarding.

Claude Mythos 5 (claude-mythos-5-*, alias claude-mythos-5-latest) is the same underlying model as Fable 5 with the safety classifiers lifted, available only through Anthropic's limited Project Glasswing release. It shares every characteristic above (context window, max output, pricing, thinking, effort, and sampling-parameter handling); the router treats both ids identically.

Claude Opus 4.8¶

Claude Opus 4.8 (claude-opus-4-8-*, alias claude-opus-4-8-latest) is the flagship Claude 4.8 model. Key characteristics:

• Context window: 1M tokens (input)
• Max output: 128K tokens
• Pricing: $5 / $25 per million tokens (input / output)
• Knowledge cutoff: January 2026
• Thinking: Adaptive only (thinking.type: "adaptive" + output_config.effort). The legacy enabled + budget_tokens shape returns HTTP 400.
• Sampling parameters: temperature, top_p, top_k are not accepted. The router drops them automatically before forwarding.
• Effort default: high. When reasoning_effort is omitted or set to auto, Anthropic applies high effort unless output_config.effort is specified.

Anthropic Fast Mode¶

Fast mode reduces latency for eligible Claude models by routing requests through Anthropic's accelerated inference path. It is gated behind a per-backend opt-in configuration flag and only applies to native Anthropic backends (never Bedrock or Vertex AI).

Enabling Fast Mode¶

Set anthropic_fast_mode: true on the backend configuration:

backends:
    - name: "anthropic-fast"
      type: anthropic
      api_key: "${CONTINUUM_ANTHROPIC_API_KEY}"
      anthropic_fast_mode: true
      models:
        - claude-opus-4-8
        - claude-opus-4-7
        - claude-opus-4-6

When anthropic_fast_mode is enabled, the router adds the anthropic-beta: fast-mode-2026-02-01 header to eligible requests.

Eligible Models¶

Fast mode applies to Opus 4.6, 4.7, and 4.8 models on native Anthropic backends. Requests to Bedrock or Vertex AI backends ignore this flag even if set.

Speed Field Passthrough¶

Clients can also request fast mode explicitly via the speed field in the request body:

{
  "model": "claude-opus-4-8",
  "speed": "fast",
  "messages": [...]
}

The speed field is forwarded on the OpenAI-compatible path. The response echoes the resolved speed in usage.speed ("fast" or "standard").

Mid-Conversation System Messages (Claude Opus 4.8+)¶

Starting with Claude Opus 4.8, the Anthropic API accepts role: "system" messages at any position in the messages array, not only at the start of the conversation. The router enables this for models that support it.

Behavior by Model Version¶

How It Works¶

When the router receives a messages array that contains role: "system" entries after a user turn, and the target model is Opus 4.8 or later, those entries are kept in place within the messages array. Leading system or developer messages before the first user turn still fill the top-level system field, because Anthropic requires at least one user message after the top-level system prompt.

For Claude 4.7 and earlier models, the prior behavior applies: all system messages are extracted and combined into the top-level system field. Mid-conversation system messages are merged into that top-level field as well, rather than preserved as in-array entries.

Example request with mid-conversation system message:

{
  "model": "claude-opus-4-8",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
    {"role": "assistant", "content": "Paris."},
    {"role": "system", "content": "Now respond only in French."},
    {"role": "user", "content": "And Germany?"}
  ]
}

For Opus 4.8+, the router sends the leading system message as the top-level system field and preserves the mid-conversation role: "system" entry in the messages array as sent to Anthropic.

OpenAI ↔ Claude Reasoning Parameter Conversion¶

The router automatically converts between OpenAI's reasoning parameters and Claude's thinking parameter, enabling cross-provider reasoning requests without client changes.

Supported OpenAI Formats:

When both formats are present, reasoning_effort (flat) takes precedence.

Effort Level to Budget Tokens Mapping:

Example Request - Chat Completions API (flat format):

// Client sends OpenAI Chat Completions API request
{
  "model": "claude-sonnet-4-6",
  "reasoning_effort": "high",
  "messages": [{"role": "user", "content": "Solve this complex problem"}]
}

// Router converts to Claude format
{
  "model": "claude-sonnet-4-6",
  "thinking": {"type": "enabled", "budget_tokens": 32768},
  "messages": [{"role": "user", "content": "Solve this complex problem"}]
}

Example Request - Responses API (nested format):

// Client sends OpenAI Responses API request
{
  "model": "claude-sonnet-4-6",
  "reasoning": {"effort": "medium"},
  "messages": [{"role": "user", "content": "Analyze this data"}]
}

// Router converts to Claude format
{
  "model": "claude-sonnet-4-6",
  "thinking": {"type": "enabled", "budget_tokens": 10240},
  "messages": [{"role": "user", "content": "Analyze this data"}]
}

Response with Reasoning Content:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "The final answer is...",
      "reasoning_content": "Let me analyze this step by step..."
    }
  }]
}

Important Notes:

• If thinking parameter is explicitly provided, it takes precedence over reasoning_effort and reasoning.effort
• reasoning_effort (flat) takes precedence over reasoning.effort (nested) when both are present
• Only models supporting extended thinking (Opus 4.x, Sonnet 4.x, Opus 4.7, Opus 4.8) will have reasoning enabled
• When reasoning is enabled, the temperature parameter is automatically removed (Claude API requirement)
• For Claude Opus 4.7 and 4.8, temperature, top_p, and top_k are always dropped regardless of thinking state
• For streaming responses, thinking content is returned as reasoning_content delta events

Environment Variables for Anthropic¶

Amazon Bedrock Backend¶

When using type: bedrock, the router routes requests through Amazon Bedrock. There are three distinct entry points:

All modes share the models: configuration surface and the same BackendTypeConfig::Bedrock variant. The split is invisible to clients, which continue to call /v1/chat/completions while the proxy adapts the body and headers. The endpoint_type field is the only knob that picks between them, with mantle as the default.

Mantle configuration¶

backends:
    - name: bedrock
      type: bedrock                      # aliases: aws-bedrock, bedrock-anthropic, AmazonBedrock
      endpoint_type: mantle              # default
      region: us-east-1                  # required; templated into the URL
      api_key: ${AWS_BEARER_TOKEN_BEDROCK}
      weight: 2
      models:
        - anthropic.claude-opus-4-7
        - us.anthropic.claude-sonnet-4-5
        - anthropic.claude-haiku-4-5
        # global.anthropic.<family> uses AWS's cheapest cross-region tier.
        # eu.<family>, jp.<family>, au.<family> route within the named geography.

Region Selection¶

The router builds the upstream URL by templating region into https://bedrock-mantle.{region}.api.aws. Any non-empty lowercase region identifier is accepted because AWS adds regions regularly — us-east-1, us-west-2, eu-west-1, ap-northeast-1, and so on. Empty or uppercase region values are rejected at configuration load time.

For Bedrock-specific forward-proxy deployments, an explicit url: field on the backend overrides the region template. Most operators should leave url unset and let the region drive the URL.

Model ID Format¶

Bedrock model identifiers come in four shapes; the router recognizes all of them and forwards them unchanged to the upstream:

Model IDs are listed explicitly in models: — there is no automatic alias mapping from native Anthropic IDs to Bedrock IDs. The router intentionally avoids hiding the geo-prefix decision behind a mapping table, since the prefix carries real billing and residency consequences.

Supported Features¶

The bedrock-mantle path inherits everything the native Anthropic backend supports:

• Streaming SSE responses, with the Anthropic SSE → OpenAI SSE transformer reused unchanged
• System prompts (translated from OpenAI's messages[role=system] to Anthropic's separate system field)
• Tool calling and tool-result round-trips
• Vision (image inputs as base64 or URLs)
• Extended thinking on Claude 4-series models, including Opus 4.7's adaptive thinking API

Authentication¶

api_key: holds the Bedrock Bearer token. Set it via an environment variable rather than a literal string in production configs:

api_key: ${AWS_BEARER_TOKEN_BEDROCK}

The router sends Authorization: Bearer ${AWS_BEARER_TOKEN_BEDROCK} on every request and strips any client-supplied x-api-key or anthropic-version header before forwarding — Bedrock returns HTTP 400 if those Anthropic-specific headers are present.

Runtime configuration (bedrock-runtime)¶

The runtime mode uses the AWS-native Invoke API at https://bedrock-runtime.{region}.amazonaws.com/model/{modelId}/invoke[-with-response-stream]. The router signs each request with SigV4 and decodes the binary application/vnd.amazon.eventstream streaming response back into OpenAI-shape SSE for clients.

Build requirement¶

The runtime path lives behind the bedrock-sigv4 Cargo feature, which pulls in the small slice of the AWS SDK needed for signing and event-stream parsing (aws-sigv4, aws-smithy-eventstream, aws-credential-types, aws-config). The default build does not include these crates. Build with the feature enabled before running a runtime backend:

cargo build --release --features bedrock-sigv4

Without the feature, configuring endpoint_type: runtime returns a clear error at startup pointing at the rebuild flag. The default mantle path works with or without the feature.

Example configuration¶

backends:
    - name: bedrock-iam
      type: bedrock
      endpoint_type: runtime               # selects the SigV4 path
      region: us-east-1
      weight: 1
      auth:
        type: sigv4                        # required for runtime
        # Pick at most one of the credential overrides below. When none
        # is set, the standard AWS chain (env, shared config, IMDS,
        # IRSA, ECS) resolves credentials.
        # aws:
        #   profile: my-bedrock-profile
        # aws:
        #   access_key_id: ${AWS_ACCESS_KEY_ID}
        #   secret_access_key: ${AWS_SECRET_ACCESS_KEY}
        #   session_token: ${AWS_SESSION_TOKEN}
      models:
          - anthropic.claude-opus-4-7
          - us.anthropic.claude-sonnet-4-5
          - global.anthropic.claude-haiku-4-5
          # Full ARNs work too (provisioned throughput, custom inference profiles):
          # - arn:aws:bedrock:us-east-1:123456789012:inference-profile/anthropic.claude-opus-4-7

Credential resolution order¶

The runtime backend resolves credentials in this order:

1. Inline static credentials under auth.aws.access_key_id + auth.aws.secret_access_key. An optional session_token covers STS-issued temporary credentials.
2. A named profile from ~/.aws/credentials and ~/.aws/config via auth.aws.profile.
3. The standard AWS chain: environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN), shared config, IMDS (EC2), IRSA / EKS pod identity, and ECS task role.

The chain resolves on every request, but the underlying AWS providers cache credentials with their own TTLs, so the per-request cost is normally a hash-map lookup rather than a network call.

Required IAM permissions¶

Attach a policy that allows bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream on the model ARNs you intend to use. A minimal example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "bedrock:InvokeModel",
                "bedrock:InvokeModelWithResponseStream"
            ],
            "Resource": [
                "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-opus-4-7",
                "arn:aws:bedrock:us-east-1:*:inference-profile/us.anthropic.claude-sonnet-4-5"
            ]
        }
    ]
}

For geo and global inference profiles, the router uses the profile ID as the URL path; AWS expands it server-side. Make sure your IAM resource list covers both the underlying foundation-model ARN and the inference-profile ARN you reference from models:.

Geo vs global profiles¶

The four model-ID shapes apply identically to runtime, but the billing and residency consequences differ between mantle and runtime only in the IAM check above. Pick the prefix that matches your data-residency and cost requirements:

Streaming details¶

Runtime streaming responses arrive as application/vnd.amazon.eventstream frames. Each chunk frame contains a base64-encoded JSON object that, once decoded, is one Anthropic SSE event (message_start, content_block_delta, etc.). The router translates this back into the OpenAI-shape SSE that clients expect, so applications that already work against the mantle path keep working unchanged. AWS-specific error frames (ThrottlingException, ValidationException, ...) are surfaced as synthetic event: error SSE chunks.

Limitations¶

• The runtime path deliberately scopes to the Invoke API. Multi-provider Bedrock models (Nova, Llama, Mistral, etc.) are reachable through the endpoint_type: converse mode below.
• The inline X-Amzn-Bedrock-GuardrailIdentifier header on Invoke is not wired through this backend. To enforce a Bedrock guardrail, use the standalone bedrock_guardrail provider in the guardrails configuration, which calls the ApplyGuardrail API independently of the model invocation and therefore works for any backend.
• Bedrock Prompt management (prompt-router ARNs) is not in scope.
• Provisioned Throughput and Application Inference Profile ARNs should work via the same URL-encoding path used for other ARNs, but no automated coverage is claimed beyond Foundation Model and inference-profile ARNs.

Converse Configuration (multi-provider)¶

The endpoint_type: converse mode targets the Converse API at https://bedrock-runtime.{region}.amazonaws.com/model/{modelId}/converse[-stream], so a single backend reaches any chat model on Bedrock through one unified wire format. It reuses the SigV4 signing, credential chain, and binary event-stream decoding from the runtime path, and therefore lives behind the same bedrock-sigv4 Cargo feature.

backends:
    - name: bedrock-converse
      type: bedrock
      endpoint_type: converse            # aliases: Converse, bedrock-converse
      region: us-east-1
      auth:
        type: sigv4                      # same auth surface as endpoint_type: runtime
      models:
        # Anthropic (also reachable via endpoint_type: runtime; listing here
        # exposes them through Converse so unified tooling works)
        - anthropic.claude-opus-4-7
        - anthropic.claude-sonnet-4-5
        # Amazon Nova
        - amazon.nova-pro-v1:0
        - amazon.nova-lite-v1:0
        - amazon.nova-micro-v1:0
        # Meta Llama
        - meta.llama4-maverick-17b-instruct-v1:0
        - meta.llama3-3-70b-instruct-v1:0
        # Mistral
        - mistral.mistral-large-2407-v1:0
        # Cohere
        - cohere.command-r-plus-v1:0
        - cohere.command-r-v1:0
        # AI21 Jamba
        - ai21.jamba-1-5-large-v1:0
        # DeepSeek
        - deepseek.deepseek-r1-v1:0

Clients keep sending standard OpenAI-shape /v1/chat/completions requests. The router translates messages into Converse content blocks (system prompts move to the top-level system array, tools become toolConfig / toolUse / toolResult, sampling parameters map into inferenceConfig), signs the request with SigV4, and translates the reply (including the binary converse-stream event stream) back into the OpenAI shape. Requests that already carry a Converse-native body (inferenceConfig, toolConfig, or a top-level system array) are detected and forwarded with minimal mutation, with no double transformation.

Per-model capability matrix¶

Bedrock models differ widely in what they accept. The router keeps a hard-coded capability table seeded from the published AWS data and rejects a request that asks for something the target model cannot do with an HTTP 4xx naming the rejected capability, before any signing or AWS round trip. Models the router does not know are never blocked; AWS stays the authority for them.

All listed families support chat, streaming, and guardrails. Geo prefixes (us., eu., apac., global., ...) and full inference-profile ARNs are normalized away before the lookup, so us.amazon.nova-pro-v1:0 matches the amazon.nova-pro row.

Vendor-specific options via additionalModelRequestFields¶

The router does not model every vendor knob. To opt into vendor-specific features, put them in a top-level extra_body object on the OpenAI request (or send a literal additionalModelRequestFields); the router forwards the fields verbatim. Examples: Claude extended thinking and Nova inference extensions.

{
  "model": "anthropic.claude-sonnet-4-5",
  "messages": [{ "role": "user", "content": "..." }],
  "extra_body": {
    "thinking": { "type": "enabled", "budget_tokens": 2048 }
  }
}

A guardrailConfig object found in extra_body is hoisted to the Converse top level where AWS expects it. Vendor-specific response extensions come back verbatim under additionalModelResponseFields.

Vision inputs¶

Image content blocks accept base64 data URLs directly. Remote http(s) image URLs are fetched by the router (SSRF-validated, size-capped) and inlined as base64, because the Converse API has no URL image source.

Claude routing precedence¶

When both the dedicated Claude path (endpoint_type: runtime) and a Converse backend list the same Claude model, routing follows the standard load-balancer semantics — there is no implicit preference. Operators who want Anthropic-native features (anthropic_beta, fine-grained thinking control) should route those models to the runtime backend explicitly; the weighted load balancer can spread traffic across both for A/B comparisons.

IAM permissions¶

Converse uses the same bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream IAM actions as the runtime path, so the policy shown above covers both modes.

Converse limitations¶

• Chat models only. Image generation (Stability, Titan Image), embeddings (Titan Embeddings, Cohere Embed), and video models are not on the Converse API and stay out of scope.
• The native Anthropic Messages endpoint (/v1/messages) is not served by converse backends; requests are rejected with a hint to use endpoint_type: runtime for native Anthropic dispatch.
• Audio input is not supported by the Converse API and is rejected with a clear error.

Native vLLM Backend¶

When using type: vllm, the router treats the backend as a vLLM OpenAI-compatible server:

• Default URL: None; the server URL must be specified (vLLM's own default is http://localhost:8000)
• Health Check: Uses /health as primary (available after model load), with /v1/models as fallback
• Model Discovery: Auto-discovers models from /v1/models; each entry's max_model_len feeds the model metadata context window when no explicit model-metadata.yaml limits are set. When several backends serve the same model id with different max_model_len values, the router publishes the minimum, the one window every backend can honor; the disagreement is logged at debug level, and explicit model-metadata.yaml or model-metadata.d/ limits still take precedence over the engine-reported minimum, so a fleet that wants a larger window advertised sets those limits explicitly or gives the larger backend a distinct model id
• Token Counting: /v1/messages/count_tokens tokenizes through the engine's /tokenize endpoint
• Passthrough: Non-standard sampling fields (top_k, min_p, repetition_penalty, ...) are forwarded unchanged
• Engine statistics: With engine_stats.enabled: true the router scrapes vLLM's GET /metrics (served on the API port, no extra server flag needed; --api-key does not guard it), plus /v1/models for context length and /version for the engine version

Minimal vLLM configuration:

backends:
  - name: "vllm-server"
    type: vllm
    url: "http://localhost:8000"
    # api_key only if the server was started with --api-key

Native SGLang Backend¶

When using type: sglang, the router provides native support for SGLang (the sglang.srt HTTP server, also the engine behind the SGLang Model Gateway). Supported aliases: SGLang, sg-lang, sg_lang.

• Default URL: http://localhost:30000 (sglang.srt default port)
• Health Check: Uses /health as primary, with /v1/models as fallback. SGLang's /health returns 503 while the engine is starting or exiting, which the router treats as warming up / unhealthy
• Model Discovery: Auto-discovers models from /v1/models; each entry's max_model_len (equal to the model's context length) feeds the model metadata context window when no explicit model-metadata.yaml limits are set. When several backends serve the same model id with different max_model_len values, the router publishes the minimum, the one window every backend can honor; the disagreement is logged at debug level, and explicit model-metadata.yaml or model-metadata.d/ limits still take precedence over the engine-reported minimum, so a fleet that wants a larger window advertised sets those limits explicitly or gives the larger backend a distinct model id
• Token Counting: /v1/messages/count_tokens tokenizes through the engine at /v1/tokenize with the vLLM-style {"model", "prompt"} payload
• Responses API: /v1/responses requests are converted to chat completions, like the rest of the vLLM family
• Passthrough: SGLang request extensions (top_k, min_p, min_tokens, regex, ebnf, repetition_penalty, lora_path, session_params, separate_reasoning, chat_template_kwargs, ...) are forwarded unchanged, and reasoning_content is preserved on messages and stream deltas, including the final usage chunk when stream_options.include_usage is set
• Authentication: Optional; when the server is started with --api-key, set api_key and the router sends Authorization: Bearer (SGLang keeps /health open even with a key)

Minimal SGLang configuration:

backends:
  - name: "sglang"
    type: sglang
    # No URL needed if using default http://localhost:30000
    # No API key required unless the server enforces one

Full SGLang configuration:

backends:
  - name: "sglang"
    type: sglang
    url: "http://192.168.1.100:30000"  # Custom URL if needed
    weight: 2
    api_key: "${SGLANG_API_KEY}"       # Optional: only if started with --api-key
    # Models are auto-discovered from /v1/models

Engine statistics: with engine_stats.enabled: true the router reads GET /v1/loads?include=core, which needs no extra server flag (a Bearer key is sent when the backend has one). Older servers without /v1/loads fall back to the deprecated GET /get_load, then to GET /metrics, which requires starting the engine with --enable-metrics. Data-parallel deployments report one entry per DP rank; the router sums counts and takes the maximum of usage fractions across ranks.

Native TensorRT-LLM Backend¶

When using type: tensorrt-llm, the router treats the backend as a TensorRT-LLM HTTP/OpenAI-compatible server. Supported aliases: TensorRTLLM, TensorRT-LLM, tensorrtllm, tensorrt_llm, trtllm, trt-llm, trt_llm.

• Default URL: None; the server URL must be specified
• Health Check: Uses /health as primary, with /v1/models as fallback
• Model Discovery: Auto-discovers models from /v1/models through the vLLM-compatible catalog parser
• Token Counting: /v1/messages/count_tokens tokenizes through the vLLM-compatible /tokenize path
• Responses API: /v1/responses requests are converted to chat completions, like the rest of the vLLM family
• KV listener salt echo: With prefix_routing.enabled: true and prefix_routing.salt_echo: true, the router injects its own cache_salt into chat-completion requests unless the client already supplied one. A continuum-kv-listener source configured with identity: salt_echo can read TensorRT-LLM blocks[].cache_salt events and attribute blocks without /detokenize. Malformed or backend-mismatched salts fall back to detokenization.

backends:
  - name: "trtllm"
    type: tensorrt-llm
    url: "http://localhost:8000"

Native llama.cpp Backend¶

When using type: llamacpp, the router provides native support for llama.cpp llama-server:

• Default URL: http://localhost:8080 (llama-server default port)
• Health Check: Uses /health endpoint (with fallback to /v1/models)
• Model Discovery: Parses llama-server's hybrid /v1/models response format
• Rich Metadata: Extracts context window, parameter count, and model size from response
• Engine statistics: With engine_stats.enabled: true the router reads GET /slots (enabled by default; --no-slots disables it) and, when the server was started with --metrics, GET /metrics. The source is selected from the endpoint_slots and endpoint_metrics capability booleans in GET /props, and default_generation_settings.n_ctx there supplies the PER-SLOT context length (the server context divided across parallel slots). A server with both endpoints disabled reports only backend_engine_stats_scrape_success 0. The same applies to type: mlxcel, degrading per endpoint it implements

Minimal llama.cpp configuration:

backends:
  - name: "local-llama"
    type: llamacpp
    # No URL needed if using default http://localhost:8080
    # No API key required for local server

Full llama.cpp configuration:

backends:
  - name: "local-llama"
    type: llamacpp
    url: "http://192.168.1.100:8080"  # Custom URL if needed
    weight: 2
    # Models are auto-discovered from /v1/models endpoint

llama.cpp Features¶

Tool Calling Auto-Detection¶

The router automatically detects tool calling capability for llama.cpp backends by querying the /props endpoint during model discovery. This enables automatic function calling support without manual configuration.

How it works:

1. When a llama.cpp backend is discovered, the router fetches the /props endpoint
2. The chat_template field is analyzed using precise Jinja2 pattern matching to detect tool-related syntax
3. If tool calling patterns are detected, the model's function_calling capability is automatically enabled
4. Detection results are stored for reference (including a hash of the chat template)

Detection Patterns:

The router uses precise pattern matching to reduce false positives:

• Role-based patterns: message['role'] == 'tool', message.role == "tool"
• Tool iteration: for tool in tools, for function in functions
• Tool calls access: .tool_calls, ['tool_calls'], message.tool_call
• Jinja2 blocks with tool keywords: {% raw %}{% if tools %}, {% for tool_call in ... %}

Example /props response analyzed:

{% raw %}

{
  "chat_template": "{% for message in messages %}{% if message['role'] == 'tool' %}...",
  "default_generation_settings": { ... },
  "total_slots": 1
}

Fallback Behavior:

• If /props is unavailable: Tool calling is assumed to be supported (optimistic fallback for modern llama.cpp versions)
• If /props returns an error: Tool calling is assumed to be supported (ensures compatibility with newer models)
• If chat template exceeds 64KB: Detection is skipped and defaults to supported
• Detection is case-insensitive for maximum compatibility
• Results are merged with any existing model metadata from model-metadata.yaml
• Detected capabilities appear in the features field of the /v1/models/{model_id} response

Model Metadata Extraction¶

The router extracts rich metadata from llama-server responses:

Starting llama-server¶

# Basic startup
./llama-server -m model.gguf --port 8080

# With GPU layers
./llama-server -m model.gguf --port 8080 -ngl 35

# With custom context size
./llama-server -m model.gguf --port 8080 --ctx-size 8192

Auto-Detection of llama.cpp Backends¶

When a backend is added without a type specified (defaults to generic), the router automatically probes the /v1/models endpoint to detect the backend type. llama.cpp backends are identified by:

1. owned_by: "llamacpp" in the response
2. Presence of llama.cpp-specific metadata fields (n_ctx_train, n_params, vocab_type)
3. Hybrid response format with both models[] and data[] arrays

This auto-detection works for:

• Hot-reload configuration changes
• Backends added via Admin API without explicit type
• Configuration files with type: generic or no type specified

Example: Auto-detected backend via Admin API:

# Add backend without specifying type - auto-detects llama.cpp
curl -X POST http://localhost:8080/admin/backends \
  -H "Content-Type: application/json" \
  -d '{
    "name": "local-llm",
    "url": "http://localhost:8080"
  }'

Native MLxcel Backend¶

When using type: mlxcel, the router provides native support for MLxcel, an MLX-based model serving backend for macOS with Apple Silicon:

• Default URL: http://localhost:8080 (same as llama-server)
• API Compatibility: Fully compatible with llama-server (llama.cpp) API
• Model Format: Serves SafeTensor format models via Apple's MLX framework
• Health Check: Uses /health as primary, with /v1/models as fallback
• Platform: macOS with Apple Silicon only

Minimal MLxcel configuration:

backends:
  - name: "mlxcel-local"
    type: mlxcel
    # No URL needed if using default http://localhost:8080

Full MLxcel configuration:

backends:
  - name: "mlxcel-local"
    type: mlxcel
    url: "http://192.168.1.100:8080"  # Custom URL if needed
    weight: 2
    models:
        - mlx-community/Qwen3-4B-4bit

Native LM Studio Backend¶

When using type: lmstudio, the router provides native support for LM Studio local server:

• Default URL: http://localhost:1234 (LM Studio default port)
• Health Check: Uses /v1/models (OpenAI-compatible) as primary, with /api/v1/models (native API) as fallback
• Model Discovery: Auto-discovers models from /v1/models endpoint
• owned_by Attribution: Reports "lmstudio" for proper model attribution
• Engine statistics: With engine_stats.enabled: true the router reads GET /api/v0/models, which needs no extra server flag; only loaded models (state: "loaded") enter the snapshot, with max_context_length as the reported context length. LM Studio has no running/waiting request count, so those series stay absent by design, not as a scrape failure

Minimal LM Studio configuration:

backends:
  - name: "lmstudio"
    type: lmstudio
    # No URL needed if using default http://localhost:1234
    # No API key required for local server

Full LM Studio configuration:

backends:
  - name: "lmstudio"
    type: lmstudio
    url: "http://192.168.1.100:1234"  # Custom URL if needed
    weight: 2
    api_key: "${LM_API_TOKEN}"        # Optional: LM Studio API token (v0.4.0+)
    # Models are auto-discovered from /v1/models endpoint

LM Studio Features¶

Native Continuum Router / Backend.AI GO Backend¶

When using type: continuum-router, the router connects to a remote Continuum Router instance or Backend.AI GO deployment for federated LLM routing. Supported aliases include: continuum-router, continuum_router, ContinuumRouter, backendai, backend-ai, backend_ai.

• Health Check: Uses /health as primary, with /v1/models as fallback
• Model Discovery: Auto-discovers models from the remote instance's /v1/models endpoint
• Authentication: Bearer token via Authorization: Bearer <key> header
• Request Passthrough: Requests are forwarded with no transformation (both systems use OpenAI-compatible APIs)
• owned_by Attribution: Reports "continuum-router" for discovered models
• Transport: Supports both HTTP and Unix Domain Socket transports

Minimal configuration:

backends:
  - name: "remote-cr"
    type: continuum-router
    url: "https://remote.example.com"
    api_key: "${REMOTE_API_KEY}"
    # Models are auto-discovered from remote /v1/models endpoint

Full configuration with explicit models:

backends:
  - name: "remote-backendai"
    type: continuum-router
    url: "https://remote-backend-ai.example.com"
    api_key: "${REMOTE_BACKEND_AI_API_KEY}"
    weight: 2
    models:
      - gpt-5.6-sol
      - claude-sonnet-4-20250514

Use cases:

• Multi-region deployment: geo-route requests across Continuum Router instances
• Federated routing: connect multiple independent CR or Backend.AI GO deployments
• Tiered access: route through a central Backend.AI GO instance for quota management
• High availability: configure multiple Backend.AI GO instances for failover

Continuum Router Backend Features¶

Unix Domain Socket Backends¶

Continuum Router supports Unix Domain Sockets (UDS) as an alternative transport to TCP for local LLM backends. Unix sockets provide:

• Enhanced Security: No TCP port exposure - communication happens through the file system
• Lower Latency: No network stack overhead for local communication
• Better Performance: Reduced context switching and memory copies
• Simple Access Control: Uses standard Unix file permissions (on Linux/macOS; Windows does not support Unix file modes)

URL Format:

unix:///path/to/socket.sock

Platform Support:

Configuration Examples:

On Windows, use drive-letter paths (e.g., unix://C:/temp/llama.sock). On Linux/macOS, use standard absolute paths (e.g., unix:///var/run/llama.sock).

backends:
  # llama-server with Unix socket (Linux/macOS)
  - name: "llama-socket"
    type: llamacpp
    url: "unix:///var/run/llama-server.sock"
    weight: 2
    models:
      - llama-3.2-3b
      - qwen3-4b

  # Ollama with Unix socket
  - name: "ollama-socket"
    type: ollama
    url: "unix:///var/run/ollama.sock"
    weight: 1
    models:
      - llama3.2
      - mistral

  # vLLM with Unix socket
  - name: "vllm-socket"
    type: vllm
    url: "unix:///tmp/vllm.sock"
    weight: 3
    models:
      - meta-llama/Llama-3.1-8B-Instruct

Starting Backends with Unix Sockets:

# llama-server
./llama-server -m model.gguf --unix /var/run/llama.sock

# Ollama
OLLAMA_HOST="unix:///var/run/ollama.sock" ollama serve

# vLLM
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B \
  --unix-socket /tmp/vllm.sock

Socket Path Conventions:

Health Checks: The router automatically performs health checks on Unix socket backends using the same endpoints (/health, /v1/models) as TCP backends.

Platform support and limits:

• Streaming: Server-Sent Events (SSE) streaming works over Unix socket backends, for both chat completions and the Anthropic Messages surface.
• Windows: AF_UNIX sockets are supported on Windows 10 1809+ (build 17063+) via the afunix.sys kernel driver; earlier Windows versions return a clear error at connect time.
• Max response size: Response bodies are limited to 100MB by default to prevent memory exhaustion.

Troubleshooting: