Server & Backends¶

Server Section¶

Controls the HTTP server behavior:

server:
  bind_address: "0.0.0.0:8080"    # Host and port to bind
  workers: 4                       # Worker threads (0 = auto)
  connection_pool_size: 100        # HTTP connection pool size

Multiple Bind Addresses and Unix Sockets¶

The server supports binding to multiple addresses simultaneously, including Unix domain sockets (on Unix-like systems and Windows 10 1809+). This enables flexible deployment scenarios such as:

• Listening on both IPv4 and IPv6 addresses
• Exposing a TCP port for external clients while using a Unix socket for local services
• Running behind a reverse proxy via Unix socket for better security

Single Address:

server:
  bind_address: "0.0.0.0:8080"

Multiple Addresses:

server:
  bind_address:
    - "127.0.0.1:8080"           # IPv4 localhost
    - "[::1]:8080"               # IPv6 localhost
    - "0.0.0.0:9090"             # All interfaces on port 9090

Unix Socket Binding (Linux, macOS, and Windows 10 1809+):

server:
  bind_address:
    - "0.0.0.0:8080"             # TCP for external access
    - "unix:/var/run/continuum-router.sock"  # Unix socket for local services
  socket_mode: 0o660              # Optional: file permissions for Unix sockets (octal)

Configuration Options:

Unix Socket Notes:

• Unix socket addresses must start with unix: prefix
• Existing socket files are automatically removed before binding
• Socket files are cleaned up on graceful shutdown
• On Windows 10 1809+ (Build 17063+), Unix sockets are fully supported via the socket2 crate
• On other non-Unix platforms, unix: addresses log a warning and are skipped
• Windows does not support Unix file permission modes; the socket_mode option is accepted but ignored
• Unix socket connections bypass IP-based authentication checks (client IP reported as "unix")

Nginx Reverse Proxy Example:

upstream continuum {
    server unix:/var/run/continuum-router.sock;
}

server {
    listen 443 ssl;
    location /v1/ {
        proxy_pass http://continuum;
    }
}

Performance Tuning:

• workers: Set to 0 for auto-detection, or match CPU cores
• connection_pool_size: Increase for high-load scenarios (200-500)

CORS Configuration¶

CORS (Cross-Origin Resource Sharing) allows the router to accept requests from web browsers running on different origins. This is essential for embedding continuum-router in:

• Tauri apps: WebView using origins like tauri://localhost
• Electron apps: Custom protocols
• Separate web frontends: Development servers on different ports

server:
  bind_address: "0.0.0.0:8080"
  cors:
    enabled: true
    allow_origins:
      - "tauri://localhost"
      - "http://localhost:*"        # Wildcard port matching
      - "https://example.com"
    allow_methods:
      - "GET"
      - "POST"
      - "PUT"
      - "DELETE"
      - "OPTIONS"
      - "PATCH"
    allow_headers:
      - "Content-Type"
      - "Authorization"
      - "X-Request-ID"
      - "X-Trace-ID"
    expose_headers:
      - "X-Request-ID"
      - "X-Fallback-Used"
    allow_credentials: false
    max_age: 3600                   # Preflight cache duration in seconds

CORS Configuration Options:

Origin Pattern Matching:

Security Considerations:

• Using * for origins allows any website to make requests - only use for public APIs
• When allow_credentials is true, you cannot use * for origins - specify exact origins
• For development, use port wildcards like http://localhost:* for flexibility
• In production, always specify exact origins for security

Reload behavior: CORS is an Axum router layer constructed at startup. Restart after changing server.cors.

Backends Section¶

Defines the LLM backends to route requests to:

backends:
  - name: "unique-identifier"        # Must be unique across all backends
    backend_id: "stable-identity"    # Optional: stable accounting identity (see below)
    type: "generic"                  # Backend type (optional, defaults to "generic")
    url: "http://backend:port"       # Base URL for the backend
    weight: 1                        # Load balancing weight (1-100)
    api_key: "${API_KEY}"            # API key (optional, supports env var references)
    org_id: "${ORG_ID}"              # Organization ID (optional, for OpenAI)
    models: ["model1", "model2"]     # Optional: exact ids or glob patterns (see "Model Globs")
    region_name: "us"                # Optional: operator-defined data-residency tag (see "Data Residency Tags")
    internal: false                  # Optional: keep infrastructure models out of user surfaces (default false)
    enabled: true                    # Optional: set false to disable user traffic while staying probeable (default true)
    http_version: auto               # Optional: auto (default), http1, or http2 (see "HTTP Protocol Version")
    responses_api: auto              # Optional: auto (default) or native (see "Native Responses API")
    replay: enabled                  # Optional: enabled (default) or disabled for a side-effecting backend (see "Replay Policy")
    anthropic_version: "2023-06-01"  # Optional, type: anthropic only: "2023-06-01" (default) or "2026-10-21" (see "Anthropic API Version")
    history_reasoning_field: as_sent # Optional: as_sent (default) or reasoning_content (see "Assistant Reasoning in History")
    supports_response_format: false  # Optional: declare response_format support; absent infers it from the type (see "Declared response_format Support")
    record_usage_missing: false      # Optional: emit a zero-token usage_missing record on 2xx responses without usage (default false)
    retry_override:                  # Optional: backend-specific retry settings
      max_attempts: 5
      initial_delay: "200ms"
      max_delay: "10s"
      backoff_multiplier: 2.0
      jitter: true
      retryable_status_codes: [429, 502, 503, 504]
      retryable_errors: [ConnectionError, TimeoutError]
      timeout: "30s"

Model Globs¶

backends[].models accepts glob entries (*) as well as exact served ids, so one entry covers a provider's whole model family without enumerating every id:

- name: openai
  type: openai
  models: ["gpt-*", "o3-*"]

A pattern is a routing rule only. /v1/models never advertises a pattern string; it lists the concrete ids the backend's own model list advertises that match the pattern, so a model the provider adds after startup appears on the next refresh without a configuration edit. A backend with no live model list (Bedrock) routes requests that match a pattern but lists nothing for it.

Resolution follows these rules:

• An exact entry wins over a pattern. When any backend lists the requested id exactly (or as a dated snapshot of an exact entry), only those backends are candidates, and a backend that matches the id only through a pattern is not.
• When no backend lists the id exactly, every backend with a matching pattern is a candidate, the same way two backends listing the same exact id share its traffic.
• Overlapping patterns on two backends (for example gpt-* and gpt-5*) load, but the router logs a warning at load and continuum-router config validate reports it at backends.<name>.models. List an id exactly on one backend to pin it there.
• An entry containing **, and a blank entry, is a load error.
• A per-key allowed_models list is matched against the requested id as before, so a pattern never widens what a key may call.
• Continuum Hub backend sync exports such a backend as preview-only (unsupported field models), because the Hub profile's allowed_models carries model ids, not patterns.

Data Residency Tags¶

region_name is an optional, operator-defined tag that marks where this backend's inference runs. It is a routing tag, not an inference: the router compares the string you wrote, so it never derives a region from the URL, the provider account, or an AWS region identifier. A tag is 1 to 64 characters from [A-Za-z0-9._-], matched exactly and case-sensitively, and an invalid value fails configuration load.

An API key with a non-empty api_keys[].allowed_regions may only route to backends whose tag is in that list. A backend without region_name is region-unknown and never satisfies such a key. When the key also sets allowed_backends, the two lists intersect; an empty intersection fails closed with 403 Forbidden instead of routing outside the requested region. The rule applies on every dispatch path, fallback hops and hedges included. See Per-Key Data-Residency Control.

Omit the field for a backend you do not want region-scoped keys to reach. This is unrelated to the Bedrock-only region field, which templates the endpoint URL.

- name: openai-eu
  type: openai
  region_name: eu
  api_key: "${OPENAI_EU_API_KEY}"
  models: ["gpt-5.6-terra"]

Endpoint URL Rule¶

A backend url is a base URL, not a full request URL: the router appends each request path (/v1/chat/completions and the rest) to it, so anything after the path silently corrupts every request the backend receives. One endpoint rule therefore applies everywhere a backend URL enters the router, from config.yaml and the Admin backend API through hub-managed backend sync and the transient backend probe:

• The value must parse as an absolute URL.
• It must not embed credentials (user:password@). backends[].url is deliberately left readable by the configuration maskers, so a credential placed there is returned in the clear by GET /admin/config/full and rendered by the WebUI, while api_key is masked.
• It must not carry a query string or a fragment, not even an empty one (a bare trailing ? or #).
• The scheme must be a transport the router can dial: http://, https://, or unix://.

The rule is enforced across all configuration entry points: startup, hot reload, the Admin backend API, hub config sync, and continuum-router config validate (which reports violations as errors and exits non-zero). Error messages name the backend and the offending property without echoing any part of the URL, preventing embedded credentials from leaking into logs or UI error surfaces. The same base validation applies to engine_stats.metrics_url, along with its additional rules (HTTP(S) schemes only, same host as the backend URL unless engine_stats.allow_external_metrics_url is set, and no HTTPS-to-HTTP downgrade).

Basic authentication for backend URLs¶

For a self-hosted engine behind a credential-checking reverse proxy, store the credential in api_key and set auth.type: basic:

- name: local-vllm
  type: vllm
  url: "http://10.0.0.5:8000/v1"
  api_key: "${VLLM_BASIC_CREDENTIAL}"   # "svc:hunter2"
  auth:
    type: basic

The router sends Authorization: Basic base64(username:password) on the wire while keeping the credential in a field the configuration maskers hide.

auth.type: basic is accepted on the self-hosted engine types and on continuumrouter: vllm, sglang, ollama, lmstudio, llamacpp, mlxcel, generic, and continuumrouter. api_key must hold username:password, split on the first colon per RFC 7617, so a password may itself contain colons. A cloud provider authenticates with its own scheme, so basic on one of those is a configuration error rather than a silently ignored field.

The configured scheme is enforced at every call site that sends the credential, backed by both visibility boundaries on the internal Bearer constructor and a compile-time source audit (tests/authorization_scheme_audit_test.rs):

• Route coverage (auth.type: basic and type: continuumrouter): type: continuumrouter connects to a remote Continuum Router or Backend.AI GO instance that may sit behind a reverse proxy requiring Basic authentication. The configured scheme applies across every credential-carrying path:
  • Model discovery and Anthropic ingress: The Unix socket model discovery path and the /anthropic/v1/messages transform path (converting Anthropic requests to OpenAI-compatible engines over HTTP or Unix sockets), as well as /anthropic/v1/messages/count_tokens.
  • Streaming chat completions: Initial dispatch, mid-stream fallback hops, auto backend selection, and Unix socket streaming. Mid-stream hops re-evaluate the scheme per hop, allowing a Bearer primary backend to fail over to a basic rescue backend.
  • Other inference and control paths: /v1/responses conversion for OpenAI-wire backends (ResponsesApiStrategy::ConvertToChatCompletions, streaming and non-streaming), ACP session/prompt dispatch, startup connection pre-warm, hub-directed synthetic probes, realtime WebSocket upgrades, and streaming image generation, image variations, and image edits (whose backend filter accepts any backend with openai in its name or URL, including a generic backend with auth.type: basic).
• OAuth on streaming routes (auth.type: oauth): Registered OAuth strategies apply to all four streaming chat-completion builders (initial dispatch, mid-stream fallback hops, auto backend selection, and Unix socket streaming) just as on non-streaming paths:
  • Tokens at or near expiry are refreshed automatically, the strategy's Authorization header is attached, and provider-specific headers (originator and the Codex User-Agent required on all ChatGPT-subscription requests) are included.
  • Mid-stream fallback hops resolve the auth strategy per hop rather than reusing the initial stream token, so token rotations, hot reloads, and cross-provider hops take effect immediately.
  • If auth.type: oauth is declared but no strategy loaded (for example, when the token store is missing or malformed), the request is sent without an Authorization header; the static api_key is never substituted. Requests for models on a ChatGPT-subscription backend are translated onto /v1/responses because that backend has no /chat/completions endpoint.
• Provider-specific schemes (service_account and sigv4): Honored only by their respective backend types (service_account on gemini, sigv4 on bedrock). If declared on any other backend type, both startup logging and continuum-router config validate emit a warning, and the backend falls back to sending its static api_key as Bearer (or no Authorization header if api_key is unset).

type: generic carries the configured credential as well. Naming a specific engine type is still recommended where one fits, because it also brings that engine's own health-check contract and its engine-statistics adapter, though generic is sufficient for authentication.

An unparseable URL is a hard load error, and an empty url on a typed backend means "use the type's default URL".

Stable Backend Identity¶

backend_id is an optional, operator-assigned identity for a backend. It matters only when the control plane is enabled: the router reports it to Continuum Hub in heartbeat inventory and stamps it on the usage records of the backend that actually served each request, which is what lets Hub charge usage to a per-backend cost center.

It exists because name cannot do the job. name is a display label an operator may rename at any hot reload, and a (provider, model) pair is not a backend, because several backends can serve one pair. Attributing cost to either would silently merge two distinct cost owners, or split one cost owner across a rename, and neither is visible after the usage has been billed.

For that reason the router never derives the value. It will not synthesize it from name, url, type, the served provider/model pair, list position, or a hash of any mutable field. You assign it, or the backend reports no identity at all.

backends:
  - name: "openai-primary"
    backend_id: "openai-eu-1"
    type: "openai"
    url: "https://api.openai.com/v1"

Rules¶

Stability¶

The value is stable across restarts because it is read from your configuration rather than minted at boot, and stable across renames because it is a separate field:

# Initial display name: charged to cost owner "openai-eu-1"
backends:
  - name: "openai-primary"
    backend_id: "openai-eu-1"

---

# Renamed display name: still charged to "openai-eu-1"
backends:
  - name: "openai-frankfurt"
    backend_id: "openai-eu-1"

A hot reload that introduces a duplicate or malformed id is rejected as a whole, and the running router keeps serving on its last known good configuration rather than adopting a partly applied identity map.

What is reported¶

The id is resolved at the moment a backend is dispatched to, so a request that fell over to a second backend is attributed to the backend that actually produced the response, not to the one selection first picked. A provider batch carries the id captured when the job was submitted, so a completion hours later still names the backend that executed it. A response served from the router's local cache dispatched no backend and therefore reports no identity. The router never substitutes the display name when an id is absent.

Heartbeat inventory reports the same ids, which is how an operator discovers which ids exist to assign: Hub holds no backend inventory of its own.

Internal (Infrastructure-Only) Backends¶

Set internal: true on a backend to keep its models out of user-facing surfaces while still serving router-internal traffic (for example a guardrail classifier, an internal summarizer, or an embeddings model). The field defaults to false (public).

When a backend is internal:

• its models are excluded from GET /v1/models (and the /v1/models/extended, single-model /v1/models/{id}, and Anthropic /anthropic/v1/models listing variants);
• a user request routed to a model served only by internal backends is rejected with model-not-found;
• router-internal callers are unaffected: a guardrail provider that names the backend via its backend: reference resolves it by name, and admin surfaces still list it.

A model served by both an internal and a public backend stays visible and routes via the public one; the internal backend name is never leaked in listings.

backends:
  - name: "guard-classifier"
    type: "vllm"
    url: "http://localhost:8001"
    internal: true
    models: ["llama-guard-3"]

Disabled Backends¶

Set enabled: false on a backend to keep it configured but out of service. The field defaults to true, so omitting it leaves existing configurations unchanged.

When a backend is disabled:

• it is still constructed in the backend pool and stays health-checked and candidate-probeable, and it remains visible on admin and inventory surfaces;
• it stays discoverable on demand through the admin discovery endpoints (POST /admin/backends/{name}/models/discover and POST /admin/backends/probe). However, it is not queried by the periodic model-list aggregation behind GET /v1/models, avoiding unnecessary connect timeouts from powered-off standby backends during background refreshes;
• it receives no request traffic at all, user or router-internal (unlike internal: true, which still serves internal callers);
• it contributes no models to the OpenAI or Anthropic model listings; a model served by both a disabled and an enabled backend stays visible and routes via the enabled one.

weight keeps its 1..=1000 range regardless of activation, so weight zero does not acquire a second meaning. Flipping enabled is hot-reloadable and takes effect without a restart.

backends:
  - name: "standby-vllm"
    type: "vllm"
    url: "http://localhost:8002"
    enabled: false
    models: ["llama-4-70b"]

HTTP Protocol Version¶

http_version chooses the HTTP protocol the router speaks to one backend: auto (the default), http1, or http2.

Why http1 exists¶

Over HTTP/2 the router multiplexes every request to one host onto a single connection, and it does not open a second connection while the first is at the server's SETTINGS_MAX_CONCURRENT_STREAMS. Request N+1 then waits inside the HTTP client until an earlier stream closes. A server that advertises 128 streams in front of generations that stream for many minutes stops accepting new requests from the router at 128 in flight, even when the engine behind it is idle.

With http_version: http1 the pool opens one connection per concurrent request, so no stream limit applies. The cost is one TCP and TLS handshake per new connection; idle connections are still kept for reuse under server.http_client.pool_idle_timeout and pool_max_idle_per_host.

backends:
  - name: "dynamo-frontend"
    type: "generic"
    url: "https://dynamo.example.internal"   # nginx ingress that negotiates h2
    http_version: http1
    models: ["solar-pro-4"]

Opening more HTTP/2 connections per host once the existing ones are full is not offered: the router's HTTP client pools one HTTP/2 connection per host and has no hook for a stream-aware second connection, so it would need a router-owned connection pool. HTTP/1.1 removes the limit without one.

Scope¶

• Covered routes: Applies to all HTTP(S) requests proxied to the backend: chat completions (streaming and non-streaming, including retries, fallback hops, and mid-stream fallback, each using its target backend's setting), completions, embeddings, the Responses API (passthrough, compact, and Chat Completions/Anthropic/Gemini conversions), Anthropic Messages and count_tokens, image generation/edit/variations, and ACP prompts.
• Client pooling: All http1 backends share one HTTP client and all http2 backends share another, each built with the same server.http_client tuning and timeouts as the default shared client. auto backends use the default shared client directly.
• Exempt internal paths: Health checks, model discovery, connection pre-warm, engine-stats scrapes, provider batch calls, and native Gemini streaming use their own dedicated clients and are unaffected. Library callers supplying a custom reqwest::Client keep it for auto backends, while http1 and http2 backends use router-built clients (logged once at warn level on startup).
• Fallback on TLS initialization failure: If the http1 or http2 client fails to build at startup, the router logs a single error line and falls back to the shared auto client until the process restarts.

Validation and hot reload¶

• Any value other than auto, http1, or http2 is a load error.
• http2 on a unix: backend is a load error: the Unix-socket transport speaks HTTP/1.1 only. http1 and auto load there and change nothing.
• The default is not written back, so an untouched configuration round-trips unchanged.
• Selection reads the backend's value from the configuration each request is routed against, so a hot reload, an Admin API backend edit, or a config-sync change applies to the next request. Requests already in flight finish on the connection they started on (gradual class).
• An http_version-only edit is reported as a backend change in the reload log and summary. The backend stays in the pool with its stats and in-flight counters.
• The Continuum Hub v1 backend profile has no protocol field, so a local export lists http_version as an unsupported field when it is not auto, which makes the entry preview-only.

Seeing a stream-limit stall¶

The router cannot observe an HTTP/2 stream-slot wait inside the client directly, so it reports what it sees from its side:

• continuum_backend_http_in_flight{backend, http_version} is the backend's in-flight request count, labelled with its configured http_version. On an auto or http2 backend a value that flattens at a round number such as 100, 128, or 250 is the server's stream limit.
• continuum_backend_response_wait_seconds{backend, http_version, outcome} is the time from handing a request to the HTTP client until its response headers arrive (outcome="response"), a transport error ends it (error), or the request is abandoned first (cancelled, for example a client that disconnected while parked). Requests queued behind a full HTTP/2 connection land in its upper buckets.
• A warn log line, at most once a minute per backend, when an http2 backend or an auto backend with an https:// URL reaches 100 requests in flight, naming http_version: http1 as the fix. An auto backend on http:// always speaks HTTP/1.1 and never warns. At debug level every dispatch logs the backend, its http_version, and its in-flight count.

See Metrics for queries.

Native Responses API¶

responses_api chooses how /v1/responses requests reach one backend: auto (the default) or native.

With auto the backend type decides, as described in the Responses API routing strategy: openai and azure pass the request through to the backend's own /v1/responses, anthropic, bedrock, and gemini convert it to their native APIs, and every other type converts it to Chat Completions. A self-hosted server that implements the Responses API itself could only be reached unchanged by declaring it type: openai, which also treats it as the commercial OpenAI provider (owned_by, credential handling, and other type-keyed behavior).

native selects pass-through for any type that speaks the OpenAI wire format and changes nothing else the type decides. Use it when the backend owns state the router cannot reproduce, for example an agent runtime whose sessions hold tool history and files, where the router's replayed transcript is not a substitute:

backends:
  - name: agent-runtime
    type: generic
    url: https://agents.example.internal
    responses_api: native
    api_key: "${AGENT_RUNTIME_KEY}"
    models:
      - research-agent

On a native backend:

• POST /v1/responses, streaming and non-streaming, is forwarded unchanged to <url>/v1/responses, including previous_response_id, conversation, and fields and input items the router does not model (for example mcp_approval_response). The backend's output, including output items the router does not model (for example mcp_approval_request), reaches the client unchanged.
• POST /v1/responses/compact is forwarded to <url>/v1/responses/compact.
• background: true is forwarded, and the result is metered and, under a guardrail, gated when it is fetched through the router; see Background responses, including the limitation for results never fetched through the router.
• GET, DELETE, and POST .../cancel on /v1/responses/{id} for a response this backend served are forwarded to it, under the checks described in Retrieve, Delete, or Cancel a Stored Response.
• The static credential is sent in the backend's configured scheme: api_key as Bearer, or Basic with auth.type: basic.
• The router does not rebuild history for previous_response_id, resolve item_reference items, or run the guardrail input stage over stored history, since the backend holds that state; the guardrail stages still check the request and the reply, as on any pass-through backend.

Droppable advisory fields¶

A conversion backend refuses every native-only /v1/responses field by default. responses_drop lets it accept an advisory one whose loss the client tolerates:

backends:
  - name: vllm-b300
    type: vllm
    url: http://10.0.0.21:8000
    responses_api: auto
    responses_drop: [include, prompt_cache_key, safety_identifier, reasoning.summary]
    models:
      - qwen3-coder

• Droppable fields: include (only when every requested value is allowlisted, today only reasoning.encrypted_content; an empty list requests nothing and is advisory), prompt_cache_key, safety_identifier, and reasoning.summary. For reasoning.summary only the summary is removed, so reasoning.mode and reasoning.context keep refusing.
• Semantic fields: text, conversation, prompt_cache_options, prompt_cache_retention, multi_agent, context_management, max_tool_calls, reasoning.mode, reasoning.context, and every tool, programmatic-calling, or input-item refusal. These always answer 400 unsupported_request_parameter, whether or not the list is set. An include value outside the allowlist is semantic too.
• Default: an empty list. With no responses_drop on the selected backend, every response is byte-for-byte the previous refusal.
• Per backend: only the selected backend's list applies. responses_api: native ignores the list entirely and drops nothing.
• Load errors: an unknown name, or a semantic field name, fails the configuration load and the error names the field.
• Disclosure: a successful reply carries x-continuum-dropped-fields with the dropped field names only, sorted and deduplicated (for example include, prompt_cache_key). The header is exposed through CORS, never carried from the client or an upstream, and is replayed on a response-cache hit.
• Metric: responses_drop_fields_total{field} increments once per dropped field per served request. A cache hit replays the header without incrementing, because no field was dropped on that request.
• Router caches: dropping prompt_cache_key does not change the router's prompt, prefix, or semantic cache keys, which are computed from the request before the drop. An opted-in request stays eligible for the response cache.
• Hot reload: the list is read from the configuration each request is routed against, so a reload applies to the next request, and an edit is reported as a backend change.

previous_response_id is not part of the list: a conversion resolves it by rebuilding history instead of dropping it.

Validation and hot reload¶

• Any value other than auto or native is a load error.
• native on anthropic, bedrock, or gemini is a load error: those types have no /v1/responses endpoint to pass through to.
• native on a unix: backend URL is a load error: Responses pass-through dispatches over HTTP(S) only.
• The default is not written back, so an untouched configuration round-trips unchanged.
• The strategy is read from the configuration each request is routed against, so a hot reload, an Admin API backend edit (responses_api on create and update), or a config-sync change applies to the next request (immediate class). A response already served keeps its binding; if its backend stops being served through pass-through, GET, DELETE, and cancel on it answer 404.
• A responses_api-only edit is reported as a backend change in the reload log and summary.
• The Continuum Hub v1 backend profile has no field for it, so a local export lists responses_api as an unsupported field when it is not auto, which makes the entry preview-only.

Replay Policy¶

replay decides whether the router may send a request it already dispatched to a backend a second time: enabled (the default) or disabled.

The router re-sends on its own in several places: the retry loop repeats a failed attempt against the same or another backend, a fallback chain moves the request to another model, the streaming relay switches backends mid-stream, and fallback.quality_check can ask a later chain entry again. For an inference backend a repeat costs tokens and nothing else. For a backend whose requests have side effects, such as an agent runtime that runs tools, it can perform the work twice: when the connection drops after the backend accepted the request, the router cannot tell whether the work happened.

backends:
  - name: "agent-runtime"
    type: "generic"
    url: "http://agent.internal:8000"
    models: ["agent:probe"]
    replay: disabled

What disabled guarantees¶

Once a dispatch to the backend could have delivered the request, any failure ends the request with that failure: a dropped connection, a timeout, and a retryable status such as 503 alike. The router then sends that invocation to no backend again:

• no retry against the same backend or a peer serving the same model, whatever retry.max_attempts and retryable_errors say;
• no hop to a fallback.fallback_chains target, whatever trigger_conditions say;
• no mid-stream switch and no pre-stream chain walk on a streaming request;
• no quality-triggered hop when the backend served the reply (the judgment still runs, and its verdict is logged and counted as skipped);
• no hop of a /v1/decisions request emulated on its chat model, after a failure or after an answer the router could not use;
• no re-dispatch through the unified fallback of disaggregated serving.

The client receives the backend's own status and body when it answered, and the router's transport error otherwise. On a streaming request the single dispatch relays an upstream 5xx as a 200 SSE stream that carries the error, where a configured chain would otherwise have walked on and answered with an HTTP error status.

Failures that occur before any request bytes leave the router are handled according to whether the transport can prove no bytes were sent:

• Pre-send connect failures on HTTP(S): On Chat Completions, Completions, embeddings, and image generation routes, a failed TCP or TLS connect to an http(s):// backend (connection refused, unreachable host, or connect timeout) is retried and falls back normally because nothing reached the backend. Router-side rejections before dispatch (open circuit breaker, saturated fallback dial bound, or backend rate limit) are likewise unaffected.
• Transports without pre-send distinction: The unix: socket transport and typed Bedrock runtime/converse dispatches do not distinguish pre-send connect errors from post-send failures, so every failure on those paths is treated as potentially delivered and stops immediately.
• Streaming and provider-shaped ingresses: A streaming request initially routed to a replay: disabled backend is dispatched once with no fallback chain walk; if a streaming fallback chain hops onto a replay: disabled backend, it stops there unless the TCP/TLS connect itself failed. Provider-shaped ingresses (/v1/responses and /anthropic/v1/messages) never hop away from a replay: disabled backend once selected, even on a refused connect.

Each client invocation is also delivered individually rather than answered from another request's execution:

• No in-flight request coalescing: Identical concurrent non-streaming requests for a model that a replay: disabled backend may serve are never coalesced. (Ordinarily, concurrent deterministic requests on /v1/chat/completions and /v1/completions with temperature: 0, /v1/embeddings, /v1/rerank, /embed_sparse, and /v1/models with identical dispatch bodies and allow-listed headers share a single upstream execution while the leader request is in flight.)
• No response caching: response_cache never stores or serves responses routed to a replay: disabled backend across any cache mode (exact, prefix, semantic, streaming replay, Responses, or Anthropic).
• No synthetic probes: Continuum Hub synthetic probes never target a replay: disabled backend, avoiding unrequested side effects.

What it does not change¶

• The backend is still selected, load-balanced, health-checked, and circuit-broken like any other. A request routed to an ordinary backend that fails can still fall back onto a replay: disabled backend; it is the attempt that reaches the side-effecting backend that is final.
• A backend answering 401 to an auth.type: oauth credential is still sent the request once more after the token refresh: a 401 is the backend's refusal to accept the request.
• The trace id the router forwards (X-Request-ID, traceparent) is a correlation id that a caller may reuse across requests, so it is not an idempotency key, and the policy does not depend on it.
• On a backend with responses_api: native, the passthrough POST /v1/responses (buffered, streamed, or background: true) is one attempt like any other Responses dispatch, so a chain stops at its failure. A queued background response is a success; the router never re-creates it, and the lifecycle calls (GET, DELETE, and POST /v1/responses/{id}/cancel) are single forwards to the bound backend with no retry or fallback.
• retry_override is not a substitute: the router does not apply it at dispatch, and it would not cover fallback chains or the streaming relay.

Validation and hot reload¶

• Any value other than enabled or disabled is a load error. The default is not written back.
• The policy is read from the configuration each request is routed against, so a hot reload, an Admin API backend edit (replay on create and update), or a config-sync change applies to the next request; a request already in flight keeps the policy it started with (immediate class).
• A replay-only edit is reported as a backend change in the reload log and summary.
• The Continuum Hub v1 backend profile has no replay field, so a local export lists replay as an unsupported field when it is disabled, which makes the entry preview-only: adopting it would silently re-enable retry and fallback.
• A backend the Hub manages (backend_config_v1) or the AppProxy worker reconciles cannot carry the field and is always enabled. In authoritative backend sync mode the Hub snapshot governs every backend not listed in immutable_backend_ids and would replace a local replay: disabled backend with a replay-enabled one, so list such a backend there.

Request Extensions¶

request_extensions lets an operator set a backend's routing preferences and attribution once, in configuration, instead of every client carrying them in every request. The motivating case is an OpenAI-compatible aggregator such as OpenRouter, which reads its routing instructions from the request body (provider, models, transforms) and its attribution from headers (HTTP-Referer, X-Title). It is a fixed JSON fragment merged into the body plus a fixed set of headers, both validated at configuration load; it is not a general request-rewriting facility, and there are no templates or expressions.

backends:
  - name: openrouter
    type: openai
    url: https://openrouter.ai/api/v1
    api_key: ${OPENROUTER_API_KEY}
    models: ["openai/gpt-5"]
    request_extensions:
      body:
        defaults:            # client wins
          provider:
            sort: throughput
        overrides:           # config wins
          provider:
            data_collection: deny
      headers:
        HTTP-Referer: https://backend.ai
        X-Title: Backend.AI GO

Merge Semantics¶

The body has two modes, mirroring request_params:

• defaults fills a key only when the client left it unset, meaning absent or JSON null. The client wins.
• overrides replaces whatever the client sent. The configuration wins. Use it for a policy a client must not be able to relax, such as provider.data_collection: deny.

Defaults apply first, then overrides. Both merge JSON objects deeply: when the client value and the configured value at the same key are both objects, the merge descends into them. Anything else is decided wholesale at that key: arrays are never merged element by element, and a type conflict (a client string where the configuration has an object) keeps the client value under defaults and takes the configured value under overrides.

For the example above, a client request carrying "provider": {"sort": "price", "data_collection": "allow"} reaches OpenRouter as:

{
  "provider": {"sort": "price", "data_collection": "deny"}
}

The client's sort survives because defaults never replaces a value the client set, and data_collection is replaced because overrides owns that one leaf. A request with no provider at all reaches the upstream with {"sort": "throughput", "data_collection": "deny"}.

Coverage¶

• Backend types: body is accepted only on backend types that send the OpenAI-compatible wire format (generic, openai, azure, vllm, sglang, tensorrt-llm, llamacpp, mlxcel, ollama, lmstudio, and continuumrouter). It is a load error on anthropic, gemini, and bedrock. headers is accepted on every backend type except bedrock with endpoint_type: runtime or converse (which sign requests via SigV4 internally and reject headers at load time); Bedrock mantle (the default endpoint type) supports headers.
• Endpoints: The body fragment applies to /v1/chat/completions, /v1/completions, /v1/responses, and /v1/embeddings, including converted ingresses that target an OpenAI-wire backend (Anthropic Messages and the Responses API converted to Chat Completions). It never applies to model listing, images, audio, files, or rerank.
• Headers: Every request that carries the backend's authentication also carries the configured extension headers:
  • Covered paths: Inference on all ingresses, model discovery (both aggregated catalog refreshes and startup auto-discovery), health probes, connection pre-warm, backend type detection, llama.cpp /props capability detection, engine-statistics scrapes, Gemini context-cache management, provider batch lifecycle calls, synthetic control-plane probes, guardrail backend: and smart-routing classifier calls, and the realtime WebSocket handshake. Backends without an api_key still receive extension headers on these paths.
  • Precedence and hot reload: A configured extension header replaces any forwarded client header of the same name, but authentication header names are rejected at load time. Header blocks are read on each send, so hot reloads, Admin API edits, and control-plane config sync apply to the next call (and restart the engine-statistics poll task when modified).
  • Admin probe and Unix-socket exceptions: During Admin backend creation, the type probe attaches candidate headers only if they already pass validation. Health probes to Unix-socket (unix:) backends omit extension headers just as they omit credentials. Because health-probe state is keyed by backend URL, two backends sharing a URL are probed using the headers of only one of them.
• Order: The body fragment is merged last into the final wire payload for the selected backend, after request-parameter policy, ingress conversion, and provider-specific normalization.

stable-diffusion.cpp Image Parameters¶

The stable-diffusion.cpp sd-server reads generation controls from a <sd_cpp_extra_args> JSON object inside prompt. Enable the typed adapter explicitly on that backend; the router does not infer the engine from a URL, backend name, or model name.

backends:
  - name: stable-diffusion-cpp
    type: generic
    url: http://localhost:7860
    models: [sd-cpp-local]
    request_extensions:
      sd_cpp_image_parameters: true

The adapter applies only to /v1/images/generations and accepts these well-typed top-level request fields:

The original fields and vendor-specific fields remain in the body. If the prompt already contains a valid first sd_cpp_extra_args object, the router merges it and gives explicit request fields precedence at matching leaves. Unrelated inline fields remain unchanged.

A malformed, unclosed, or non-object first block is forwarded unchanged, and the router does not append another block. Generated JSON escapes < so a string such as negative_prompt cannot terminate the tag early. Each retry and fallback hop starts from the original body, which prevents an opted-in backend's prompt from reaching an ordinary backend.

This adapter is accepted only on OpenAI-wire backend types. It does not make static request_extensions.body apply to images and does not change extension-header coverage.

Fallback and Retries¶

Extensions are applied per selected backend at dispatch, on every retry and every fallback hop, including pre-stream and mid-stream streaming fallback. A hop that moves to a different backend starts from the unextended request and carries that backend's extensions, or none, and never the previous backend's. Configure the block on every backend in a chain that needs it.

Validation¶

Every rule below is enforced at configuration load on every ingestion path (startup, hot reload, the Admin config and backend APIs, config import, control-plane sync, continuum-router config validate, and the MCP validate tool). Errors name the exact path, for example backends[2].request_extensions.body.overrides.temperature.

Reserved body keys, compared case-insensitively at the top level of defaults and overrides:

Why these body keys are reserved, and which keys remain settable:

• Session and output-shape keys: background (Responses) returns a queued response requiring store: true rather than an immediate result. include and include_reasoning add output fields (reasoning.encrypted_content, message.output_text.logprobs). modalities with "audio" and audio (Chat) add choices[].message.audio to the response, and echo (legacy /v1/completions) prepends the prompt to choices[].text. guided_json, guided_regex, guided_choice, guided_grammar, and structured_outputs (vLLM 0.12+) constrain output shape like response_format, while prompt_logprobs adds prompt-side log probabilities alongside logprobs.
• Per-caller and prompt keys: prompt_cache_key is a per-caller cache key (an operator-wide default would collapse all callers into one bucket), safety_identifier is a per-end-user identity like user, and prediction (Predicted Outputs) comes from the caller's prompt like messages.
• Settable generation and account keys: Sampling parameters belong in request_params (which validates types and ranges). However, stop, seed, and logit_bias are settable in request_extensions because they steer generation without altering the response schema (except on Codex OAuth backends, which strip stop). Account-level options such as service_tier and prompt_cache_retention / prompt_cache_options are also settable. Nested keys under an allowed top-level key are free-form, and null is rejected anywhere in the fragment.

Reserved header names are compared case-insensitively with every _ normalized to - (since CGI-style proxies and servers fold X_Api_Key and X-Api-Key together; X_Title alongside X-Title is also rejected as a duplicate). Because the router and its auth layer manage these headers directly, extensions cannot set or override:

• Standard HTTP and transport headers: Authorization, Proxy-Authorization, Host, Content-Length, Content-Type, Content-Encoding, Transfer-Encoding, Connection, Keep-Alive, Proxy-Connection, Upgrade, TE, Trailer, Expect, Cookie, Set-Cookie, Accept, Accept-Encoding, User-Agent, and any sec-websocket-* header.
• Provider and authentication headers: x-api-key, api-key, x-goog-api-key, x-auth-token, x-access-token, anthropic-version, anthropic-beta, OpenAI-Organization, OpenAI-Project, chatgpt-account-id, originator, and any x-amz-* or x-amzn-* header.
• Tracing and loop-detection headers: X-Request-ID, X-Trace-ID, X-Correlation-ID, traceparent, tracestate, baggage, x-continuum-router-hops, and any custom header name configured under tracing.headers.
• Router-internal attribution header: x-continuum-resolved-user-profile-id.

anthropic-user-profile-id is accepted as a backend's static Anthropic user profile for a single-tenant backend. The name must be spelled with hyphens, the value must be uprof_ followed by 1 to 128 ASCII letters, digits, _, or -, the header is refused on bedrock backends, and an API key's own profile replaces it on the wire. See Per-Customer Attribution for Anthropic.

Bounds:

A header value outside SP-~ (café, Hangul, a non-breaking space) is refused at load rather than sent as obs-text: HeaderValue::from_str at dispatch accepts bytes 0x80-0xFF, and RFC 9110 treats that byte range as obsolete, so upstreams and intermediaries handle it inconsistently, some reject the request and others read it as Latin-1. Percent-encode non-ASCII text instead. The error message never contains the header value.

Secrets¶

Header values support ${ENV_VAR} interpolation exactly like api_key. Every value under headers is treated as a secret whatever the header is called, because an aggregator key can travel under any header name: values are masked on the Admin config and backend reads, in YAML/TOML/JSON export, in config history, and in the WebUI, while header names stay visible. Sending a masked document back through PUT /admin/config/backends or PUT /admin/backends/{name} keeps the stored value. config diff and the MCP tools redact the values, extension header values are never logged, and config show --resolved prints them in plaintext exactly as it prints api_key. Body fragment values are not secrets and are shown as configured.

Hot Reload and Continuum Hub¶

• Live dispatch and health probes: Changes to request_extensions take effect on the next request without a restart. Publishing a new configuration updates the backend's dispatch block, refreshes health-probe headers, and revalidates the model list.
• Response cache namespace: A backend with a non-empty body fragment scopes its response-cache entries under a fingerprint of that fragment. Editing defaults or overrides on reload causes a cache miss rather than serving responses generated under the old fragment. Extension headers do not affect the cache key, and backends without a body fragment use standard cache keys.
• In-flight request deduplication: The retry layer coalesces concurrent deterministic non-streaming requests that share the same dispatch body and deduplication allow-list headers while a leader request is in flight. Because deduplication is scoped to the active published configuration, any reload starts a fresh deduplication scope.
• Continuum Hub compatibility: The Continuum Hub v1 backend profile does not include request_extensions. Hub-owned backends never carry the block, hub sync preserves the block on surviving local backends, and local backend exports mark entries with request_extensions as preview-only (listing the field name as unsupported without exporting its values). In authoritative backend sync mode, add any local backend whose request_extensions must survive Hub snapshots to immutable_backend_ids.

One Leading System Message¶

Some chat templates accept at most one system message, and only at index 0 of the messages array. For example, Qwen/Qwen3.8-* templates raise System message must be at the beginning. whenever a system message appears after the first turn, causing vLLM to return HTTP 400 with that message, which the router relays to the client. Three common request patterns trigger this template error:

• A mid-conversation system message, which Claude Code sends from turn 2 onward via /anthropic/v1/messages (and which the Anthropic ingress preserves as role: "system").
• A second system message placed immediately after the first.
• A client-supplied mid-array system message on /v1/chat/completions, which passthrough routing normally forwards unchanged.

Because this constraint belongs to the model's chat template rather than the inference engine, it is configured via the requires_single_leading_system model metadata key:

• Default catalog coverage: The shipped model-metadata.yaml enables requires_single_leading_system for the Qwen3.8 family (qwen3.8-27b, qwen3.8-2.4t-a95b, qwen3.8-flash-next). Hugging Face model IDs such as Qwen/Qwen3.8-27B-FP8 match those entries automatically through namespace-prefix stripping and quantization-suffix normalization across all OpenAI-wire backend types (vllm, sglang, generic, openai, etc.).
• Per-backend or drop-in configuration: If your deployment runs without model-metadata.yaml or uses a custom model name, set requires_single_leading_system: true in the backend's model_configs entry (or in a model-metadata.d/ drop-in):

backends:
  - name: qwen-engine
    type: vllm
    url: http://localhost:8003
    model_configs:
      - id: qwen38-27b-fp8
        metadata:
          requires_single_leading_system: true

Unlike metadata fields that replace an entry wholesale, requires_single_leading_system resolves field-wise across sources: a model_configs entry that omits the key (for example, one added only to override pricing) inherits the catalog value, and only an explicit requires_single_leading_system: false disables it (useful when running an engine with a custom --chat-template).

When requires_single_leading_system is active for a model, the router normalizes /v1/chat/completions payloads as follows:

• Merging system messages: All system entries in the payload are concatenated in order, separated by a blank line (\n\n), into a single leading system message while all other messages keep their relative order. Payloads with no system entry or a single system entry already at index 0 are left untouched.
• Multimodal and structured content: Text-part arrays contribute their text to the merged message; non-text parts (such as images) are preserved in order within a combined parts array. If a system entry contains content that is neither a string nor a parts array, the router forwards the payload unchanged.

For example, this input array:

[
  {"role": "system", "content": "You are a coding assistant."},
  {"role": "user", "content": "Read README.md and summarize it."},
  {"role": "assistant", "content": "Here is the summary."},
  {"role": "system", "content": "Be brief from now on."},
  {"role": "user", "content": "Now the second file."}
]

reaches the backend as:

[
  {"role": "system", "content": "You are a coding assistant.\n\nBe brief from now on."},
  {"role": "user", "content": "Read README.md and summarize it."},
  {"role": "assistant", "content": "Here is the summary."},
  {"role": "user", "content": "Now the second file."}
]

• Dispatch scope and hot reload: The merge runs at every /v1/chat/completions dispatch site carrying a client conversation (OpenAI, Anthropic Messages, Responses, and ACP ingresses) and is evaluated per fallback hop against that hop's backend and model (so a mid-stream relay hop starts from the original unmerged array). Changes via hot reload, Admin API edits, config sync, or metadata drop-ins apply to the next request. Native anthropic, gemini, and bedrock dispatches and single-turn control-plane probes are unaffected.
• Trade-off: Merging preserves system role priority instead of downgrading instructions to user text, but mid-conversation instructions move to the front of the prompt and apply from turn 1. For this reason, the behavior is enabled per model rather than globally.

Assistant Reasoning in History¶

Clients that replay an assistant turn's reasoning send it under one of two field names: reasoning_content (what the router itself returns) or reasoning (what newer vLLM returns). vLLM reads both. A Dynamo frontend reads only reasoning_content and drops reasoning before the prompt is rendered, so the model never sees that reasoning.

history_reasoning_field: reasoning_content moves the field for one backend:

backends:
  - name: solar-pro4
    type: vllm
    url: http://solar-pro4-frontend:8000
    history_reasoning_field: reasoning_content
    models:
      - solar-pro4

On such a backend, each assistant message in messages is rewritten as follows:

• String reasoning: moves to reasoning_content.
• Both fields present: reasoning_content is kept and reasoning is removed, so the backend sees one field. A null or empty reasoning_content counts as absent, so the string reasoning replaces it.
• null reasoning: removed.
• Any other reasoning value (an object or an array): sent unchanged, because reasoning_content is a string field.

Messages of other roles are not touched.

Validation and hot reload¶

• The values are as_sent (the default, which sends every message as the client wrote it) and reasoning_content. Any other value is a load error.
• reasoning_content on anthropic, bedrock, or gemini is a load error, since those types convert the messages array away.
• The rewrite runs at the same /v1/chat/completions dispatch sites as the system-message merge, per fallback hop, so a hop to a backend without the field sends the history as the client wrote it.
• The field is read from the configuration each request is routed against, so a hot reload or an Admin API backend edit (history_reasoning_field on create and update) applies to the next request (immediate class).
• The Continuum Hub v1 backend profile has no field for it, so a local export lists history_reasoning_field as an unsupported field when it is not as_sent.
• A backend the Hub manages (backend_config_v1) cannot carry the field and is always as_sent. In authoritative backend sync mode the Hub snapshot governs every backend not listed in immutable_backend_ids and would return a local reasoning_content backend to as_sent, so list such a backend there.

Declared response_format Support¶

On /v1/chat/completions and /v1/completions, the router routes a request carrying response_format only to backends that carry the field to their provider, and answers 400 unsupported_request_parameter when the requested model has none. By default it infers this from the backend type. An OpenAI-compatible engine is inferred as carrying the field because the router forwards the raw body, even when the engine itself rejects or ignores json_schema, such as an older llama.cpp build.

supports_response_format states what the engine does:

backends:
  - name: old-llamacpp
    type: llamacpp
    url: http://llamacpp:8080
    supports_response_format: false
    models:
      - local-model

Finding undeclared backends¶

continuum_response_format_inferred_dispatch_total{backend} counts, once per request, each candidate retained for a response_format request on inference rather than a declaration. A request this check refuses is not counted; a request refused later, for example by a guardrail, or answered from the response cache is. A backend with a growing count is one to test and declare.

Validation and hot reload¶

• supports_response_format: true on anthropic or bedrock loads, but the router logs a warning at load and continuum-router config validate reports it at backends.<name>.supports_response_format.
• The field is read from the configuration each request is routed against, so a hot reload or an Admin API backend edit applies to the next request (immediate class). On Admin API update, clear_supports_response_format: true removes the declaration.
• The Continuum Hub v1 backend profile has no field for it, so a local export lists supports_response_format as an unsupported field when it is set.
• A backend the Hub manages (backend_config_v1) or the AppProxy worker reconciles cannot carry the field and is always inferred. In authoritative backend sync mode, list a backend with a declaration in immutable_backend_ids to keep it.

Model Artifact Identity¶

The same model id is often served from different weights across backends (different quantizations, Hugging Face revisions, or re-exported checkpoints). The artifact and draft_model metadata keys identify the exact weights a backend serves, stamping the resolved model_digest and draft_model onto every usage record for that backend. Shared declarations typically live in a model-metadata.d/ drop-in (see Model artifact identity), while a backend serving a distinct build overrides it in model_configs:

backends:
  - name: qwen-fp8
    type: vllm
    url: http://localhost:8003
  - name: qwen-int4
    type: vllm
    url: http://localhost:8004
    model_configs:
      - id: qwen3.8-27b
        metadata:
          artifact:
            digest: sha256:0b0b000000000000000000000000000000000000000000000000000000000002
            registry_version: v2.1-int4

• Per-backend precedence: Overrides apply per backend, one key at a time (artifact and draft_model resolve independently). In the example above, qwen-int4 reports its own artifact digest while inheriting draft_model from the drop-in, whereas qwen-fp8 uses the drop-in's artifact. An overriding artifact replaces the shared artifact object as a whole rather than merging sub-fields.
• Usage records and API surfaces: Usage records carry the digest of the backend that actually served the request, including after a retry or fallback hop. GET /admin/models lists every serving backend's identity under artifacts, while GET /v1/models/{model} displays the first serving backend's identity in metadata.
• Validation and public source safety: A malformed digest in model_configs fails configuration loading, and config validate warns when a vllm, sglang, or tensorrt-llm entry declares a draft_model without digests. Because artifact.source is exposed publicly on both /v1/models/{model} and /admin/models, never use presigned or token-bearing URLs (?token=, ?sig=); use a stable public locator such as hf://org/model or oci://registry/repo@sha256:....

Starting Without Backends¶

The router can start with an empty backends list (backends: []), which is useful for:

• Infrastructure bootstrapping: Start the router first, then add backends dynamically via the Admin API
• Container orchestration: Router container can be ready before backend services
• Development workflows: Test admin endpoints before backends are provisioned
• Gradual rollout: Start with zero backends and add them progressively

When running with no backends:

• /v1/models returns {"object": "list", "data": []}
• /v1/chat/completions and other routing endpoints return 503 "No backends available"
• /health returns healthy status (the router itself is operational)
• Backends can be added via POST /admin/backends

Example minimal configuration for dynamic backend management:

server:
  bind_address: "0.0.0.0:8080"

backends: []  # Start with no backends - add via Admin API later

health_checks:
  interval: "30s"
  timeout: "5s"
  unhealthy_threshold: 3
  healthy_threshold: 2
  endpoint: "/health"

logging:
  level: info
  format: json

admin:
  auth:
    method: bearer
    bearer_token: "${ADMIN_TOKEN}"

Backend Types Supported:

Native OpenAI Backend¶

When using type: openai, the router provides: - Default URL: https://api.openai.com/v1 (can be overridden for proxies) - Built-in model metadata: Automatic pricing, context windows, and capabilities - Environment variable support: Automatically loads from CONTINUUM_OPENAI_API_KEY and CONTINUUM_OPENAI_ORG_ID

Minimal OpenAI configuration:

backends:
  - name: "openai"
    type: openai
    models:
      - gpt-5.6-sol
      - gpt-5.6-terra
      - o3-mini

Full OpenAI configuration with explicit API key:

backends:
  - name: "openai-primary"
    type: openai
    api_key: "${CONTINUUM_OPENAI_API_KEY}"
    org_id: "${CONTINUUM_OPENAI_ORG_ID}"     # Optional
    models:
      - gpt-5.6-sol
      - gpt-5.6-terra
      - o1
      - o1-mini
      - o3-mini
      - text-embedding-3-large

Using OpenAI with a proxy:

backends:
  - name: "openai-proxy"
    type: openai
    url: "https://my-proxy.example.com/v1"   # Override default URL
    api_key: "${PROXY_API_KEY}"
    models:
      - gpt-5.6-sol

ChatGPT subscription / Codex headless login (OAuth device flow)¶

This section covers Codex as an upstream for the router's own clients. To point the Codex CLI itself at the router, see Integrating with Codex CLI.

Continuum Router can authenticate against the OpenAI Codex backend (https://chatgpt.com/backend-api/codex) using a ChatGPT Plus / Pro / Enterprise subscription rather than a paid OpenAI API key.

The OpenAI provider does not implement standards-compliant RFC 8628 device flow; instead, the router uses OpenAI's custom Codex headless device-code flow, which is what the official Codex CLI uses for "login on headless devices." The flow has three steps:

1. Request a one-time user_code from auth.openai.com/api/accounts/deviceauth/usercode.
2. Poll auth.openai.com/api/accounts/deviceauth/token until the user approves the code in their browser.
3. Exchange the resulting authorization code for access / refresh tokens via PKCE at auth.openai.com/oauth/token.

Every request carries an originator: codex_cli_rs header so the Cloudflare front in front of auth.openai.com admits the traffic.

One-time login¶

Run the device-flow login from any machine that can open the OpenAI verification URL in a browser. The router prints a verification URL and a short user code, and polls the token endpoint until the device is approved.

continuum-router auth login --backend openai-chatgpt

Tokens are written atomically to the configured token_store with mode 0600 on Unix. After login, the router uses these tokens automatically and refreshes them transparently before they expire (60-second clock-skew margin). A 401 response from the backend triggers one forced refresh and a single retry before the error surfaces to the client.

Backend configuration¶

backends:
  - name: openai-chatgpt
    type: openai
    url: https://chatgpt.com/backend-api/codex
    auth:
      type: oauth
      oauth:
        provider: openai
        token_store: ~/.continuum-router/auth/openai.json
    # Codex backends enumerate their real models from the live endpoint (see
    # "Dynamic model enumeration" below). A non-empty list selects which
    # enumerated models to expose and is the fallback when enumeration fails or
    # returns nothing; leave it empty to expose every enumerated model.
    models:
      - gpt-5
      - codex-mini

client_id and scope default to the public Codex CLI values that auth.openai.com accepts; you only need to override them for a custom OAuth client registration.

Configuration reference¶

Dynamic model enumeration¶

Unlike static OpenAI API backends, the ChatGPT Codex backend does not expose a standard /v1/models endpoint. Instead it serves a model list at GET <base>/models?client_version=<ver> that is gated by the account's subscription plan. The router queries this endpoint with the loaded OAuth token during model discovery (and on the normal refresh cadence) and uses the result to populate /v1/models and routing:

• The request carries Authorization: Bearer <access_token>, the originator: codex_cli_rs header, and a chatgpt-account-id header derived from the id_token.
• Only user-facing models survive: an entry is kept when its visibility is list and its available_in_plans either is empty or contains the account plan (read from the id_token's chatgpt_plan_type claim). Internal entries such as codex-auto-review (visibility hide) are never exposed.
• The configured models: selection is applied to the enumerated set, exactly like every other backend. A non-empty list intersects the enumerated models down to the operator-selected subset; an empty list exposes the full enumerated set (e.g. gpt-5.5, gpt-5.4, gpt-5.4-mini). Enumerated models carry the clean owner openai instead of the raw backend name.
• On any failure (network error, non-2xx, empty list), the router falls back to the configured models: list so routing degrades gracefully. Dynamic enumeration applies only to Codex OAuth backends; other OAuth and static-key backends use their configured model lists.

Request handling¶

The ChatGPT Codex backend serves a single inference endpoint, <base>/responses (https://chatgpt.com/backend-api/codex/responses for the default base), which implements a stricter subset of the public OpenAI Responses API. The router converts /v1/chat/completions requests routed to a Codex OAuth backend into Responses-API form and adjusts the converted request to the shape Codex accepts:

• input is always sent as an item list; Codex rejects the bare-string shorthand that a single-message request would otherwise produce.
• max_output_tokens, temperature, top_p, presence_penalty, frequency_penalty, and stop are removed, because Codex rejects them.
• A tool_choice that forces a specific function is downgraded to "required", with a warning in the log. Codex supports only the string modes, so the model is still pushed to call a tool without naming one.
• When the request carries no system message, the instructions field is filled with a minimal default, because Codex rejects requests without instructions.

Codex accepts only stream: true with store: false, and the router forces both on every converted upstream call:

• Streaming clients: Receive the converted SSE stream as usual.
• Non-streaming clients (stream: false): The router consumes the upstream SSE stream and folds it into a single chat-completion JSON body before responding (including Codex responses that omit the text/event-stream content type).
• Stateless storage: Because store is always false, nothing is persisted on the OpenAI side, and conversation state travels in the request as with any Chat Completions client.

Operational notes¶

• Tokens never appear in logs, traces, or metrics; only a short redacted prefix is logged when a refresh occurs.
• Refreshes are single-flighted with a tokio::sync::Mutex, so concurrent in-flight requests during expiry windows do not stampede the OAuth provider.
• Static api_key configurations are unaffected; OAuth is opt-in per backend via the auth.type: oauth block.
• Re-running continuum-router auth login --backend <name> rewrites the token store atomically and is safe while the router is running.
• The token store is read leniently: expires_at accepts epoch seconds (the canonical form the router writes), a numeric string, or an RFC3339 datetime, so a token file produced by another tool loads without editing.
• The OpenAI provider's auth endpoint is Cloudflare-protected; the default user_agent and originator: codex_cli_rs header mirror the official Codex CLI so the device flow reaches the OAuth endpoint instead of the bot-challenge page. Set auth.oauth.user_agent and/or auth.oauth.originator to custom values only if your environment specifically requires them.
• OAuth is rendered as oauth in YAML; o_auth is also accepted as an alias.

Environment Variables for OpenAI¶

Model discovery¶

The aggregated model service refreshes live backend catalogs and caches the result. For ordinary HTTP backends it requests the OpenAI-compatible model-list path derived from the backend URL; Unix-socket backends use /v1/models.

• A non-empty backends[].models list is an allowlist applied to the discovered result.
• Native Anthropic: Supports live model discovery via GET {base}/v1/models using x-api-key and the backend's anthropic-version (2023-06-01 unless anthropic_version says otherwise; over HTTP or Unix domain sockets, defaulting an empty url to https://api.anthropic.com) with cursor pagination (has_more / after_id), and falls back to the router's curated Claude model catalog if live discovery fails. Anthropic backends do not require an explicit models list at validation time; when models or allowed_models uses dateless or -latest canonical IDs (such as claude-haiku-4-5 or claude-haiku-4-5-latest), the router matches them against date-suffixed live Anthropic model IDs (such as claude-haiku-4-5-20251001).
• Amazon Bedrock: Has no model-list endpoint and requires an explicit models or model_configs list; aggregation serves Bedrock straight from the configured list without making HTTP /v1/models requests.
• Backends with enabled: false are skipped entirely: the aggregation makes no request to them and they contribute no models. Use the admin discovery endpoints when you need a disabled backend's live catalog.
• A 404 or 405 from a model-list endpoint is treated as permanent, like 401/403: the refresh fails that backend immediately instead of retrying a provider that plainly serves no model list. 5xx responses and connection errors still use the full retry budget.
• OAuth backends other than the ChatGPT Codex flow use their configured models list. A Codex-flow backend enumerates its account-visible models from its authenticated Codex model endpoint and then applies the configured allowlist when non-empty.
• A failed backend is omitted from that refresh. If every backend fails, the refresh returns and caches an empty model list. Ordinary OpenAI-compatible discovery failures have no hardcoded fallback catalog (native Anthropic uses its curated Claude catalog fallback described above).
• model_configs supplies backend metadata and matching information; it is not a replacement for a required explicit models list on backends that do not support live discovery.

On-demand and administrative discovery endpoints:

• POST /v1/models/refresh or POST /admin/models/refresh: Triggers an on-demand refresh of the aggregated catalog (the public route is rate-limited).
• POST /admin/backends/{name}/models/discover: Fetches the live catalog from one exact backend before the configured models: allowlist is applied. This endpoint is read-only and does not change models:, model aliases, health state, circuit-breaker state, or the aggregated model cache. Codex OAuth backends use the same account- and plan-aware discovery path as normal aggregation, while unsupported backend types return a machine-readable model_discovery_unsupported error instead of falling back to configured model names.
• POST /admin/backends/probe: Validates a candidate URL/key and populates an initial model picker before registration (advertised via /admin/capabilities as transient_backend_probe_v1):
  • Uses the same backend type, URL composition, auth headers, parser, normalization, response-size cap, and model-count limit as registered discovery.
  • Stays request-local and never writes to configuration, history, pools, health state, circuit state, caches, environment files, or token stores. Configured fallbacks accept models or model_configs and remain subject to the per-backend model-count limit.
  • Reports health.credential_status separately from catalog.source so an unauthenticated health success cannot be mistaken for credential validity; transient OAuth health is explicitly unknown because its lifecycle-managed authentication strategy is unavailable before registration.

Upstream vendor and metadata¶

An OpenAI-compatible aggregator such as OpenRouter lists hundreds of models the router has no model-metadata.yaml entry for, and its catalog is the only source of their vendor, context window, output cap, and price. Discovery reads those fields from each /v1/models entry as follows.

Vendor (owned_by), resolved per entry against the normalized model id:

1. A real upstream owned_by is kept unchanged.
2. When the upstream sends no owned_by at all (absent, null, or blank) and the id has a vendor/model namespace, the first path segment is the vendor, exactly as sent. OpenRouter sends no owned_by, so anthropic/claude-haiku-4.5 reports anthropic and openrouter/auto reports openrouter. A segment that is itself a placeholder value (such as vllm/...) is not used.
3. Otherwise a placeholder (system, organization_owner, an empty value on an id without a namespace, and so on) is replaced with the backend type's owner: type: openai reports openai, and types without a canonical owner such as generic keep the value.

The rule reads only the id's first path segment, so a path-like id with no owned_by reports that segment even when it is not a vendor (hf.co/org/model reports hf.co). An explicit placeholder is not the same as a missing value: OpenAI's own catalog sends owned_by: "system", which still becomes openai. A per-model owned_by in model-metadata.yaml or backend model_configs overrides the result, and response_defaults.owned_by applies only to a value that remains a placeholder, so a namespace-derived vendor outranks that global default.

• Context window and output cap:
  • Context length is read from max_model_len (vLLM, SGLang) or context_length (OpenRouter); when both are present, max_model_len wins. It fills limits.context_window when the effective metadata has no limits, within 1..=10000000.
  • limits.max_output is set to top_provider.max_completion_tokens when present and within 1..=context_length, and to the context length otherwise.
  • When several backends serve one ID, both values are the minimum across backends.
• Pricing:
  • pricing.prompt and pricing.completion are read as USD per token (JSON string or number) and published as pricing.input_tokens and pricing.output_tokens in USD per 1M tokens (for example, "0.000001" becomes 1.0). Both must be valid or no price is published.
  • Any component that does not parse to a finite, non-negative number (including OpenRouter's "-1" for a variable price) or exceeds 100,000 USD per 1M tokens means no price rather than zero.
  • pricing.input_cache_read becomes cached_input_discount (1 - cache_read / prompt) when that ratio is between 0 and 1. Other pricing keys (input_cache_write, web_search) and non-object pricing shapes are ignored. Published prices feed smart-routing tier inference and cost scoring just like configured prices.
• Precedence for limits: Configured metadata wins. Upstream context window and output cap fill limits only when the metadata matched from model-metadata.yaml, model-metadata.d/, backend model_configs, or the built-in OpenAI catalog has no limits.
• Precedence for pricing:
  • Because pricing is provider-specific, an upstream price wins over shared model-metadata.yaml (including model-metadata.d/ drop-ins) and the built-in OpenAI catalog regardless of how the entry matched (exact ID, alias, date/quantization normalization, or vendor/ prefix stripping). This prevents a shipped 0/0 self-hosted entry from making an aggregator's minimax/minimax-m2 appear free.
  • Only a backend-specific price in backends[].model_configs overrides that backend's upstream price.
  • When at least one backend serving the ID reports a usable upstream price, each backend contributes either its model_configs pin or its upstream price (backends with neither, or with "-1", are excluded). Across candidates, the router publishes the maximum of each price component and the minimum cached_input_discount (any candidate without a discount clears it). When no backend reports a usable upstream price, pricing falls back to configured metadata.
• Other metadata fields: Every other metadata field (including owned_by) keeps configured metadata first. An upstream metadata object is never trusted as router metadata.
• Where it appears:
  • GET /v1/models: Keeps its OpenAI field set (id, object, created, owned_by); only owned_by changes.
  • GET /v1/models/extended: Carries the mapped metadata and passes all other upstream fields through unchanged (pricing, architecture, supported_parameters, top_provider, canonical_slug, ...), except router-owned keys (id, object, created, owned_by, backends, metadata, tier, domains), which are never duplicated, and max_model_len / context_length, which are overwritten with metadata.limits.context_window when present so the entry states a single window (see List Models with Metadata).
  • GET /v1/models/{model}: Carries the mapped metadata and max_tokens. A namespaced ID's / works raw or percent-encoded (/v1/models/anthropic/claude-haiku-4.5 and /v1/models/anthropic%2Fclaude-haiku-4.5 are equivalent). The segments extended, proxy, and refresh are reserved for their own routes.

Native Gemini Backend¶

When using type: gemini, the router provides:

• Default URL: https://generativelanguage.googleapis.com/v1beta/openai (OpenAI-compatible endpoint)
• Automatic Dual-API routing: Uses the OpenAI-compatible endpoint for standard chat requests and automatically switches to the Gemini Native API (/v1beta/models/{model}:generateContent and :streamGenerateContent?alt=sse) when Gemini-exclusive features are requested or when configured to prefer the native API
• Built-in model metadata: Automatic context windows and capabilities for Gemini models
• Environment variable support: Automatically loads from CONTINUUM_GEMINI_API_KEY
• Extended streaming timeout: 300s timeout for thinking models (gemini-3.1-pro, gemini-3-flash, gemini-2.5-pro)
• Automatic max_tokens adjustment: For thinking models, see below

Minimal Gemini configuration:

backends:
  - name: "gemini"
    type: gemini
    models:
      - gemini-3.1-pro-preview
      - gemini-3-flash-preview
      - gemini-2.5-pro
      - gemini-2.5-flash
      - nano-banana-2
      - nano-banana-pro

Full Gemini configuration with API Key:

backends:
  - name: "gemini"
    type: gemini
    api_key: "${CONTINUUM_GEMINI_API_KEY}"
    weight: 2
    models:
      - gemini-3.1-pro-preview
      - gemini-3-flash-preview
      - gemini-2.5-pro
      - gemini-2.5-flash
      - gemini-2.5-flash-image
      - nano-banana
      - nano-banana-2
      - nano-banana-2-lite
      - nano-banana-pro

Gemini Dual-API Routing and Exclusive Features¶

A type: gemini backend supports both Google's OpenAI-compatible surface (/v1beta/openai/chat/completions) and the Gemini Native API (/v1beta/models/{model}:generateContent and :streamGenerateContent?alt=sse authenticated with x-goog-api-key or OAuth Bearer token):

• Automatic per-request switching: Standard /v1/chat/completions requests use the OpenAI-compatible endpoint by default. When a request includes any Gemini-exclusive feature, the router automatically converts it to the native generateContent format and normalizes the response (and SSE stream) back to OpenAI format:
  • Google Search grounding (tools with google_search / web_search, or web_search_options)
  • Python Code Execution (tools with code_execution / code_interpreter)
  • Native image output (modalities: ["text", "image"], image_config, or a Gemini image generation model)
  • Multi-turn assistant images history
  • extra_body.google native fields (grounding, response_modalities, safety_settings, media_resolution, cached_content)
• Always prefer the Gemini Native API: To route all /v1/chat/completions traffic for Gemini through the native generateContent / streamGenerateContent endpoints (even when no exclusive tool is present), either set gemini_context_cache.prefer_native_api: true globally or set the backend url to the native base (https://generativelanguage.googleapis.com/v1beta):

gemini_context_cache:
  enabled: false
  prefer_native_api: true

Gemini Image Models (Nano Banana family):

Gemini Authentication Methods¶

The Gemini backend supports two authentication methods:

API Key Authentication (Default)¶

The simplest authentication method using a Google AI Studio API key:

backends:
  - name: "gemini"
    type: gemini
    api_key: "${CONTINUUM_GEMINI_API_KEY}"
    models:
      - gemini-3.1-pro-preview

Service Account Authentication¶

For enterprise environments and Google Cloud Platform (GCP) deployments, you can use Service Account authentication with automatic OAuth2 token management:

backends:
  - name: "gemini"
    type: gemini
    auth:
      type: service_account
      key_file: "/path/to/service-account.json"
    models:
      - gemini-3.1-pro-preview
      - gemini-3-flash-preview

Using environment variable for key file path:

backends:
  - name: "gemini"
    type: gemini
    auth:
      type: service_account
      key_file: "${GOOGLE_APPLICATION_CREDENTIALS}"
    models:
      - gemini-3.1-pro-preview

Service Account Authentication Features:

Creating a Service Account Key:

1. Go to Google Cloud Console
2. Navigate to IAM & Admin > Service Accounts
3. Create a new service account or select an existing one
4. Click Keys > Add Key > Create new key
5. Choose JSON format and download the key file
6. Store the key file securely and reference it in your configuration

Required Permissions:

The service account needs the following roles for Gemini API access:

• roles/aiplatform.user - For Vertex AI Gemini endpoints
• Or appropriate Google AI Studio permissions for generativelanguage.googleapis.com

Authentication Priority¶

When multiple authentication methods are configured:

If both api_key and auth are specified, the auth block takes precedence and a warning is logged.

Gemini Thinking Models: Automatic max_tokens Adjustment¶

Gemini "thinking" models (gemini-3.1-pro, gemini-3-flash, gemini-2.5-pro, and models with -pro-preview suffix) perform extended reasoning before generating responses. To prevent response truncation, the router automatically adjusts max_tokens:

This ensures thinking models can generate complete responses without truncation due to low default values from client libraries.

Environment Variables for Gemini¶

Native Anthropic Backend¶

When using type: anthropic, the router provides:

• Default URL: https://api.anthropic.com (can be overridden for proxies)
• Native API translation: Automatically converts OpenAI format requests to Anthropic Messages API format and vice versa
• Anthropic-specific headers: Automatically adds x-api-key and anthropic-version headers (the version is the backend's anthropic_version; see "Anthropic API Version")
• Environment variable support: Automatically loads from CONTINUUM_ANTHROPIC_API_KEY
• Extended streaming timeout: 600s timeout for extended thinking models (Claude Opus, Sonnet 4)

Minimal Anthropic configuration:

backends:
  - name: "anthropic"
    type: anthropic
    models:
      - claude-sonnet-5
      - claude-haiku-4-5

Full Anthropic configuration:

backends:
  - name: "anthropic"
    type: anthropic
    api_key: "${CONTINUUM_ANTHROPIC_API_KEY}"
    weight: 2
    anthropic_fast_mode: false       # opt-in: enable fast mode for eligible models (default false)
    models:
      - claude-fable-5
      - claude-mythos-5   # limited release (Project Glasswing); needs approved access
      - claude-opus-4-8
      - claude-opus-4-7
      - claude-opus-4-6
      - claude-sonnet-4-6
      - claude-haiku-4-5

Anthropic API Version¶

anthropic_version selects the anthropic-version header the router sends to a type: anthropic backend: 2023-06-01 (the default when the field is absent) or 2026-10-21.

backends:
  - name: "anthropic"
    type: anthropic
    api_key: "${CONTINUUM_ANTHROPIC_API_KEY}"
    anthropic_version: "2026-10-21"
    models:
      - claude-sonnet-5

Quote the value. In YAML both spellings parse, but in TOML an unquoted date is a date literal, not a string.

Where the version is sent¶

The setting applies to every request the router originates toward the backend:

• OpenAI-format requests (/v1/chat/completions, buffered and streaming) and /v1/responses converted to the Messages API, over HTTP and Unix sockets.
• The native /anthropic/v1/messages and /anthropic/v1/messages/count_tokens ingresses, only when the client sent no anthropic-version. A client value is forwarded verbatim in both directions: a client that sends 2023-06-01 is not upgraded, and a client that sends 2026-10-21 reaches a default backend with that version.
• Health checks (a health_check.headers entry named anthropic-version still wins), live model discovery, connection pre-warm, Continuum Hub probes, and Message Batch calls.

Outbound authentication stays x-api-key. The API accepts both x-api-key and Authorization: Bearer, and a Foundry-hosted endpoint requires x-api-key, so switching would gain nothing. Inbound, the router accepts the 2026-10-21 SDKs' Authorization: Bearer <router key> on /anthropic/v1/messages exactly like x-api-key: the same authentication, allow-lists, rate-limit bucket, and usage attribution.

Request changes under 2026-10-21¶

Under 2026-10-21 Anthropic refuses three request shapes with HTTP 400. The router adjusts the bodies it converts from the OpenAI and Responses formats:

Native /anthropic/v1/messages requests also drop client sampling fields on Anthropic hops that forbid them. The client version takes precedence over the backend version, and models that reject sampling under every version also drop these fields. Other provider hops retain their client sampling values; Bedrock behavior is unchanged.

Sampling values from request_params¶

A request_params or Continuum Hub sampling value that would be sent to a 2026-10-21 backend is refused with a router-authored 400 before dispatch, on the OpenAI, Responses, and native Messages paths alike, rather than being reported as applied and then dropped. This matters for models that still honor sampling under 2023-06-01 (Claude Opus and Sonnet below 4.7, Haiku 4.5).

Upstream offers no replacement, so remove those fields for the backend's models or keep 2023-06-01 on a backend that serves them. continuum-router config validate warns at backends[N].anthropic_version when a listed model is affected, or when the policy sets a sampling field globally and the backend lists no models.

Validation and hot reload¶

• Values other than 2023-06-01 and 2026-10-21 are a load error, and so is the field on any other backend type, including bedrock, whose body version (bedrock-2023-05-31) has no documented 2026 equivalent. The error names backends[N].anthropic_version.
• The version is read from the configuration each request is routed against, so a hot reload, an Admin API backend edit (anthropic_version on create and update, clear_anthropic_version to remove it), or a config-sync change applies to the next request (immediate class). The pooled backend that serves probes is rebuilt on the same reload.
• The Continuum Hub v1 backend profile has no version field, so a local export lists anthropic_version as an unsupported field when it is set.

Anthropic API Translation¶

The router automatically handles the translation between OpenAI and Anthropic API formats:

When Anthropic returns stop_reason: "refusal", the router maps it to the OpenAI-compatible finish_reason: "content_filter". The upstream stop_details object (containing the refusal category) is forwarded on the response choice under stop_details and omitted when absent.

Example Request Translation:

OpenAI format (incoming from client):

{
  "model": "claude-sonnet-4-5-20250929",
  "messages": [
    {"role": "system", "content": "You are helpful."},
    {"role": "user", "content": "Hello"}
  ],
  "max_tokens": 1024
}

Anthropic format (sent to API):

{
  "model": "claude-sonnet-4-5-20250929",
  "system": "You are helpful.",
  "messages": [
    {"role": "user", "content": "Hello"}
  ],
  "max_tokens": 1024
}

Anthropic Native API Endpoints¶

In addition to routing OpenAI-format requests to Anthropic backends, the router also provides native Anthropic API endpoints:

These endpoints allow clients that use Anthropic's native API format (such as Claude Code) to connect directly without any request/response transformation overhead.

Claude Code Compatibility¶

The Anthropic Native API endpoints include full compatibility with Claude Code and other advanced Anthropic API clients:

Prompt Caching Support:

The router preserves cache_control fields throughout the request/response pipeline:

• System prompt text blocks
• User message content blocks (text, image, document)
• Tool definitions
• Tool use and tool result blocks

Header Forwarding:

Cache Usage Reporting:

Streaming responses from native Anthropic backends include cache usage information:

{
  "usage": {
    "input_tokens": 2159,
    "cache_creation_input_tokens": 2048,
    "cache_read_input_tokens": 0
  }
}

Anthropic's input_tokens excludes both cache buckets. Across protocol bridges, the router maps cache usage to each target API's convention:

• Forward bridge (Anthropic/Bedrock -> /v1/chat/completions): Follows the OpenAI convention. prompt_tokens equals input_tokens + cache_read_input_tokens + cache_creation_input_tokens, prompt_tokens_details.cached_tokens carries cache reads, and top-level cache_read_input_tokens and cache_creation_input_tokens are kept. Bedrock Converse is mapped the same way (inputTokens + cacheReadInputTokens + cacheWriteInputTokens).
• Reverse bridge (OpenAI/Responses -> /anthropic/v1/messages): Follows the Anthropic convention:
  • input_tokens is reported as the upstream prompt count minus cache reads (prompt_tokens_details.cached_tokens, input_tokens_details.cached_tokens, or top-level cache_read_input_tokens) and minus cache writes.
  • Both buckets are returned as cache_read_input_tokens and cache_creation_input_tokens in the non-streaming body and in the streaming message_start/message_delta usage alike (a usage block that is already Anthropic-shaped passes through unchanged).
  • Clients that sum input_tokens + cache_read_input_tokens + cache_creation_input_tokens (such as context-window calculators) count cached prompt tokens once across non-streaming bodies and bridged streams (including local cache replay and TokenUsage::from_json_value re-parsing final message_delta values).
  • Downstream routers that meter a bridged stream with a native Anthropic streaming tracker also record the exact total: because message_start on a bridged stream carries input_tokens: 0 and real counts arrive in message_delta, the tracker reads input_tokens, cache_read_input_tokens, and cache_creation_input_tokens from message_delta as well.
• Responses API bridge (Anthropic -> /v1/responses): Follows the Responses API convention. input_tokens is the inclusive prompt count, and input_tokens_details (sibling output_tokens_details; singular input_token_details is also accepted on input) carries cached_tokens (cache reads) and cache_write_tokens (cache writes) on both the non-streaming body and the stream's response.completed event.

Tool-Call ID Normalization:

tool_use.id is scoped to the whole conversation in the Anthropic protocol, but some OpenAI-compatible backends behind this endpoint only guarantee an id unique within a single response (vLLM's Kimi-K3 template restarts its {tool_name}:{index} counter on every response). The router rewrites an outbound id that isn't already known to be conversation-unique to crt_<nonce:12><base64url_nopad(raw_id)> and restores the original bytes when the client replays it, so tool_use/tool_result pairing stays correct across turns with no configuration required. Ids already unique across the conversation (the toolu_, call_, fc_, and chatcmpl-tool- prefixes) pass through untouched.

Mid-Conversation System Messages:

A role: "system" message after the first user turn (the Opus 4.8+ shape, which Claude Code sends from its second turn on) is forwarded to an OpenAI-compatible backend at the system role, in place, so the instruction keeps system priority. A model whose chat template accepts only one leading system entry rejects that; the router merges the entries for models whose metadata sets requires_single_leading_system, which the catalog carries for the Qwen3.8 family.

Anthropic Extended Thinking Models¶

Models supporting extended thinking (Claude Opus, Sonnet 4, Claude Opus 4.7, Claude Opus 4.8, Claude Sonnet 5, Claude Haiku 4.5, and Claude Haiku 5.5) may require longer response times. The router automatically:

• Sets higher default max_tokens (16384) for thinking models
• Uses extended streaming timeout (600s) for these models

For Claude Opus 4.7/4.8, Claude Sonnet 5.5, Claude Haiku 5.5, and Mythos-class models (Fable 5, Mythos 5):

• Adaptive thinking required: These models require thinking.type: "adaptive" with output_config.effort and reject the legacy budget_tokens shape. The router normalizes explicit legacy thinking.type: "enabled" requests to adaptive thinking automatically.
• Sampling parameters dropped: temperature, top_p, and top_k are not accepted and are dropped automatically.
• Explicit disabled thinking: Fable 5, Mythos 5, and Sonnet 5.5 reject an explicit thinking.type: "disabled" (HTTP 400); the router omits the thinking parameter entirely for these models.

Claude Sonnet 5.5¶

Claude Sonnet 5.5 (claude-sonnet-5-5) uses adaptive thinking with effort levels low through max; reasoning_effort: "xhigh" maps to max. It rejects explicit disabled thinking and forced tool_choice values any and tool. The router omits the disabled-thinking field and rejects a forced choice before dispatch. auto and none tool choices remain valid.

Claude Fable 5¶

Claude Fable 5 (claude-fable-5-*, alias claude-fable-5-latest) is Anthropic's most capable model, a Mythos-class flagship positioned a tier above Opus 4.8. Key characteristics:

• Context window: 1M tokens (input)
• Max output: 128K tokens
• Pricing: $10 / $50 per million tokens (input / output)
• Thinking: Adaptive only (thinking.type: "adaptive" + output_config.effort). The legacy enabled + budget_tokens shape returns HTTP 400, and an explicit thinking.type: "disabled" also returns HTTP 400 (the router omits the thinking parameter instead).
• Effort: Supports low, medium, high, and max (reasoning_effort: "xhigh" maps to max).
• Sampling parameters: temperature, top_p, top_k are not accepted. The router drops them automatically before forwarding.

Claude Mythos 5 (claude-mythos-5-*, alias claude-mythos-5-latest) is the same underlying model as Fable 5 with the safety classifiers lifted, available only through Anthropic's limited Project Glasswing release. It shares every characteristic above (context window, max output, pricing, thinking, effort, and sampling-parameter handling); the router treats both ids identically.

Claude Opus 4.8¶

Claude Opus 4.8 (claude-opus-4-8-*, alias claude-opus-4-8-latest) is the flagship Claude 4.8 model. Key characteristics:

• Context window: 1M tokens (input)
• Max output: 128K tokens
• Pricing: $5 / $25 per million tokens (input / output)
• Knowledge cutoff: January 2026
• Thinking: Adaptive only (thinking.type: "adaptive" + output_config.effort). The legacy enabled + budget_tokens shape returns HTTP 400.
• Sampling parameters: temperature, top_p, top_k are not accepted. The router drops them automatically before forwarding.
• Effort default: high. When reasoning_effort is omitted or set to auto, Anthropic applies high effort unless output_config.effort is specified.

Claude Haiku 5.5¶

Claude Haiku 5.5 (claude-haiku-5-5) is Anthropic's fastest model, for high-volume, latency-sensitive tasks. The id is fixed, with no date suffix and no -latest alias. Key characteristics:

• Context window: 1M tokens (input)
• Max output: 128K tokens
• Pricing: $0.10 / $0.50 per million tokens (input / output) for prompts up to 100,000 tokens, and $0.50 / $2.50 for prompts over 100,000 tokens
• Thinking: Adaptive, on by default. The legacy enabled + budget_tokens shape returns HTTP 400, so the router converts it to adaptive thinking. An explicit thinking.type: "disabled" is accepted.
• Effort: Supports low through max (reasoning_effort: "xhigh" maps to max). Anthropic's default is medium.
• Sampling parameters: temperature, top_p, top_k are not accepted. The router drops them automatically before forwarding.

Claude Haiku 4.5 (claude-haiku-4-5-*) keeps manual extended thinking: the router converts reasoning_effort into a thinking.budget_tokens config, as it does for Sonnet 4.x.

Anthropic Fast Mode¶

Fast mode reduces latency for eligible Claude models by routing requests through Anthropic's accelerated inference path. It is gated behind a per-backend opt-in configuration flag and only applies to native Anthropic backends (never Bedrock or Vertex AI).

Enabling Fast Mode¶

Set anthropic_fast_mode: true on the backend configuration:

backends:
    - name: "anthropic-fast"
      type: anthropic
      api_key: "${CONTINUUM_ANTHROPIC_API_KEY}"
      anthropic_fast_mode: true
      models:
        - claude-opus-4-8
        - claude-opus-4-7
        - claude-opus-4-6

When anthropic_fast_mode is enabled, the router adds the anthropic-beta: fast-mode-2026-02-01 header to eligible requests.

Eligible Models¶

Fast mode applies to Opus 4.6, 4.7, and 4.8 models on native Anthropic backends. Requests to Bedrock or Vertex AI backends ignore this flag even if set.

Speed Field Passthrough¶

Clients can also request fast mode explicitly via the speed field in the request body:

{
  "model": "claude-opus-4-8",
  "speed": "fast",
  "messages": [...]
}

The speed field is forwarded on the OpenAI-compatible path. The response echoes the resolved speed in usage.speed ("fast" or "standard").

Mid-Conversation System Messages (Claude Opus 4.8+)¶

Starting with Claude Opus 4.8, the Anthropic API accepts role: "system" messages at any position in the messages array, not only at the start of the conversation. The router enables this for models that support it.

Behavior by Model Version¶

How It Works¶

When the router receives a messages array that contains role: "system" entries after a user turn, and the target model is Opus 4.8 or later, those entries are kept in place within the messages array. Leading system or developer messages before the first user turn still fill the top-level system field, because Anthropic requires at least one user message after the top-level system prompt.

For Claude 4.7 and earlier models, all system messages are extracted and combined into the top-level system field. Mid-conversation system messages are merged into that top-level field as well, rather than preserved as in-array entries.

Example request with mid-conversation system message:

{
  "model": "claude-opus-4-8",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
    {"role": "assistant", "content": "Paris."},
    {"role": "system", "content": "Now respond only in French."},
    {"role": "user", "content": "And Germany?"}
  ]
}

For Opus 4.8+, the router sends the leading system message as the top-level system field and preserves the mid-conversation role: "system" entry in the messages array as sent to Anthropic.

OpenAI ↔ Claude Reasoning Parameter Conversion¶

The router automatically converts between OpenAI's reasoning parameters and Claude's thinking parameter, enabling cross-provider reasoning requests without client changes.

Supported OpenAI Formats:

When both formats are present, reasoning_effort (flat) takes precedence.

Effort Level to Budget Tokens Mapping:

Example Request - Chat Completions API (flat format):

// Client sends OpenAI Chat Completions API request
{
  "model": "claude-sonnet-4-6",
  "reasoning_effort": "high",
  "messages": [{"role": "user", "content": "Solve this complex problem"}]
}

// Router converts to Claude format
{
  "model": "claude-sonnet-4-6",
  "thinking": {"type": "enabled", "budget_tokens": 32768},
  "messages": [{"role": "user", "content": "Solve this complex problem"}]
}

Example Request - Responses API (nested format):

// Client sends OpenAI Responses API request
{
  "model": "claude-sonnet-4-6",
  "reasoning": {"effort": "medium"},
  "messages": [{"role": "user", "content": "Analyze this data"}]
}

// Router converts to Claude format
{
  "model": "claude-sonnet-4-6",
  "thinking": {"type": "enabled", "budget_tokens": 10240},
  "messages": [{"role": "user", "content": "Analyze this data"}]
}

Response with Reasoning Content:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "The final answer is...",
      "reasoning_content": "Let me analyze this step by step..."
    }
  }]
}

Important Notes:

• If thinking parameter is explicitly provided, it takes precedence over reasoning_effort and reasoning.effort
• reasoning_effort (flat) takes precedence over reasoning.effort (nested) when both are present
• Only models supporting extended thinking (Opus 4.x, Sonnet 4.x, Opus 4.7, Opus 4.8, Haiku 4.5, Haiku 5.5) will have reasoning enabled
• When reasoning is enabled, the temperature parameter is automatically removed (Claude API requirement)
• For Claude Opus 4.7, Opus 4.8, and Haiku 5.5, temperature, top_p, and top_k are always dropped regardless of thinking state
• For streaming responses, thinking content is returned as reasoning_content delta events

Environment Variables for Anthropic¶

Amazon Bedrock Backend¶

When using type: bedrock, the router routes requests through Amazon Bedrock. There are three distinct entry points:

All modes share the models: configuration surface and the same BackendTypeConfig::Bedrock variant. The split is invisible to clients, which continue to call /v1/chat/completions while the proxy adapts the body and headers. The endpoint_type field is the only knob that picks between them, with mantle as the default.

Mantle configuration¶

backends:
    - name: bedrock
      type: bedrock                      # aliases: aws-bedrock, bedrock-anthropic, AmazonBedrock
      endpoint_type: mantle              # default
      region: us-east-1                  # required; templated into the URL
      api_key: ${AWS_BEARER_TOKEN_BEDROCK}
      weight: 2
      models:
        - anthropic.claude-opus-4-7
        - us.anthropic.claude-sonnet-4-5
        - anthropic.claude-haiku-4-5
        # global.anthropic.<family> uses AWS's cheapest cross-region tier.
        # eu.<family>, jp.<family>, au.<family> route within the named geography.

Region Selection¶

The router builds the upstream URL by templating region into https://bedrock-mantle.{region}.api.aws. Any non-empty lowercase region identifier is accepted because AWS adds regions regularly (us-east-1, us-west-2, eu-west-1, ap-northeast-1, and so on). Empty or uppercase region values are rejected at configuration load time.

For Bedrock-specific forward-proxy deployments, an explicit url: field on the backend overrides the region template. Most operators should leave url unset and let the region drive the URL.

Model ID Format¶

Bedrock model identifiers come in four shapes; the router recognizes all of them and forwards them unchanged to the upstream:

Model IDs are listed explicitly in models:; there is no automatic alias mapping from native Anthropic IDs to Bedrock IDs. The router intentionally avoids hiding the geo-prefix decision behind a mapping table, since the prefix carries real billing and residency consequences.

Supported Features¶

The bedrock-mantle path inherits everything the native Anthropic backend supports:

• Streaming SSE responses, with the Anthropic SSE → OpenAI SSE transformer reused unchanged
• System prompts (translated from OpenAI's messages[role=system] to Anthropic's separate system field)
• Tool calling and tool-result round-trips
• Vision (image inputs as base64 or URLs)
• Extended thinking on Claude 4-series models, including Opus 4.7's adaptive thinking API

Authentication¶

api_key: holds the Bedrock Bearer token. Set it via an environment variable rather than a literal string in production configs:

api_key: ${AWS_BEARER_TOKEN_BEDROCK}

The router sends Authorization: Bearer ${AWS_BEARER_TOKEN_BEDROCK} on every request and strips any client-supplied x-api-key or anthropic-version header before forwarding (Bedrock returns HTTP 400 if those Anthropic-specific headers are present).

Runtime configuration (bedrock-runtime)¶

The runtime mode uses the AWS-native Invoke API at https://bedrock-runtime.{region}.amazonaws.com/model/{modelId}/invoke[-with-response-stream]. The router signs each request with SigV4 and decodes the binary application/vnd.amazon.eventstream streaming response back into OpenAI-shape SSE for clients.

Build requirement¶

The runtime path lives behind the bedrock-sigv4 Cargo feature, which pulls in the small slice of the AWS SDK needed for signing and event-stream parsing (aws-sigv4, aws-smithy-eventstream, aws-credential-types, aws-config). The default build does not include these crates. Build with the feature enabled before running a runtime backend:

cargo build --release --features bedrock-sigv4

Without the feature, configuring endpoint_type: runtime returns a clear error at startup pointing at the rebuild flag. The default mantle path works with or without the feature.

Example configuration¶

backends:
    - name: bedrock-iam
      type: bedrock
      endpoint_type: runtime               # selects the SigV4 path
      region: us-east-1
      weight: 1
      auth:
        type: sigv4                        # required for runtime
        # Pick at most one of the credential overrides below. When none
        # is set, the standard AWS chain (env, shared config, IMDS,
        # IRSA, ECS) resolves credentials.
        # aws:
        #   profile: my-bedrock-profile
        # aws:
        #   access_key_id: ${AWS_ACCESS_KEY_ID}
        #   secret_access_key: ${AWS_SECRET_ACCESS_KEY}
        #   session_token: ${AWS_SESSION_TOKEN}
      models:
          - anthropic.claude-opus-4-7
          - us.anthropic.claude-sonnet-4-5
          - global.anthropic.claude-haiku-4-5
          # Full ARNs work too (provisioned throughput, custom inference profiles):
          # - arn:aws:bedrock:us-east-1:123456789012:inference-profile/anthropic.claude-opus-4-7

Credential resolution order¶

The runtime backend resolves credentials in this order:

1. Inline static credentials under auth.aws.access_key_id + auth.aws.secret_access_key. An optional session_token covers STS-issued temporary credentials.
2. A named profile from ~/.aws/credentials and ~/.aws/config via auth.aws.profile.
3. The standard AWS chain: environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN), shared config, IMDS (EC2), IRSA / EKS pod identity, and ECS task role.

The chain resolves on every request, but the underlying AWS providers cache credentials with their own TTLs, so the per-request cost is normally a hash-map lookup rather than a network call.

Required IAM permissions¶

Attach a policy that allows bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream on the model ARNs you intend to use. A minimal example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "bedrock:InvokeModel",
                "bedrock:InvokeModelWithResponseStream"
            ],
            "Resource": [
                "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-opus-4-7",
                "arn:aws:bedrock:us-east-1:*:inference-profile/us.anthropic.claude-sonnet-4-5"
            ]
        }
    ]
}

For geo and global inference profiles, the router uses the profile ID as the URL path; AWS expands it server-side. Make sure your IAM resource list covers both the underlying foundation-model ARN and the inference-profile ARN you reference from models:.

Geo vs global profiles¶

The four model-ID shapes apply identically to runtime, but the billing and residency consequences differ between mantle and runtime only in the IAM check above. Pick the prefix that matches your data-residency and cost requirements:

Streaming details¶

Runtime streaming responses arrive as application/vnd.amazon.eventstream frames. Each chunk frame contains a base64-encoded JSON object that, once decoded, is one Anthropic SSE event (message_start, content_block_delta, etc.). The router translates this back into the OpenAI-shape SSE that clients expect, so applications that already work against the mantle path keep working unchanged. AWS-specific error frames (ThrottlingException, ValidationException, ...) are surfaced as synthetic event: error SSE chunks.

Limitations¶

• The runtime path deliberately scopes to the Invoke API. Multi-provider Bedrock models (Nova, Llama, Mistral, etc.) are reachable through the endpoint_type: converse mode below.
• The inline X-Amzn-Bedrock-GuardrailIdentifier header on Invoke is not wired through this backend. To enforce a Bedrock guardrail, use the standalone bedrock_guardrail provider in the guardrails configuration, which calls the ApplyGuardrail API independently of the model invocation and therefore works for any backend.
• Bedrock Prompt management (prompt-router ARNs) is not in scope.
• Provisioned Throughput and Application Inference Profile ARNs should work via the same URL-encoding path used for other ARNs, but no automated coverage is claimed beyond Foundation Model and inference-profile ARNs.

Converse Configuration (multi-provider)¶

The endpoint_type: converse mode targets the Converse API at https://bedrock-runtime.{region}.amazonaws.com/model/{modelId}/converse[-stream], so a single backend reaches any chat model on Bedrock through one unified wire format. It reuses the SigV4 signing, credential chain, and binary event-stream decoding from the runtime path, and therefore lives behind the same bedrock-sigv4 Cargo feature.

backends:
    - name: bedrock-converse
      type: bedrock
      endpoint_type: converse            # aliases: Converse, bedrock-converse
      region: us-east-1
      auth:
        type: sigv4                      # same auth surface as endpoint_type: runtime
      models:
        # Anthropic (also reachable via endpoint_type: runtime; listing here
        # exposes them through Converse so unified tooling works)
        - anthropic.claude-opus-4-7
        - anthropic.claude-sonnet-4-5
        # Amazon Nova
        - amazon.nova-pro-v1:0
        - amazon.nova-lite-v1:0
        - amazon.nova-micro-v1:0
        # Meta Llama
        - meta.llama4-maverick-17b-instruct-v1:0
        - meta.llama3-3-70b-instruct-v1:0
        # Mistral
        - mistral.mistral-large-2407-v1:0
        # Cohere
        - cohere.command-r-plus-v1:0
        - cohere.command-r-v1:0
        # AI21 Jamba
        - ai21.jamba-1-5-large-v1:0
        # DeepSeek
        - deepseek.deepseek-r1-v1:0

Clients keep sending standard OpenAI-shape /v1/chat/completions requests. The router translates messages into Converse content blocks (system prompts move to the top-level system array, tools become toolConfig / toolUse / toolResult, sampling parameters map into inferenceConfig), signs the request with SigV4, and translates the reply (including the binary converse-stream event stream) back into the OpenAI shape. Requests that already carry a Converse-native body (inferenceConfig, toolConfig, or a top-level system array) are detected and forwarded with minimal mutation, with no double transformation.

Per-model capability matrix¶

Bedrock models differ widely in what they accept. The router keeps a hard-coded capability table seeded from the published AWS data and rejects a request that asks for something the target model cannot do with an HTTP 4xx naming the rejected capability, before any signing or AWS round trip. Models the router does not know are never blocked; AWS stays the authority for them.

All listed families support chat, streaming, and guardrails. Geo prefixes (us., eu., apac., global., ...) and full inference-profile ARNs are normalized away before the lookup, so us.amazon.nova-pro-v1:0 matches the amazon.nova-pro row.

Vendor-specific options via additionalModelRequestFields¶

The router does not model every vendor knob. To opt into vendor-specific features, put them in a top-level extra_body object on the OpenAI request (or send a literal additionalModelRequestFields); the router forwards the fields verbatim. Examples: Claude extended thinking and Nova inference extensions.

{
  "model": "anthropic.claude-sonnet-4-5",
  "messages": [{ "role": "user", "content": "..." }],
  "extra_body": {
    "thinking": { "type": "enabled", "budget_tokens": 2048 }
  }
}

A guardrailConfig object found in extra_body is hoisted to the Converse top level where AWS expects it. Vendor-specific response extensions come back verbatim under additionalModelResponseFields.

Vision inputs¶

Image content blocks accept base64 data URLs directly. Remote http(s) image URLs are fetched by the router (SSRF-validated, size-capped) and inlined as base64, because the Converse API has no URL image source.

Claude routing precedence¶

When both the dedicated Claude path (endpoint_type: runtime) and a Converse backend list the same Claude model, routing follows the standard load-balancer semantics without implicit preference. Operators who want Anthropic-native features (anthropic_beta, fine-grained thinking control) should route those models to the runtime backend explicitly; the weighted load balancer can spread traffic across both for A/B comparisons.

IAM permissions¶

Converse uses the same bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream IAM actions as the runtime path, so the policy shown above covers both modes.

Converse limitations¶

• Chat models only. Image generation (Stability, Titan Image), embeddings (Titan Embeddings, Cohere Embed), and video models are not on the Converse API and stay out of scope.
• The native Anthropic Messages endpoint (/v1/messages) is not served by converse backends; requests are rejected with a hint to use endpoint_type: runtime for native Anthropic dispatch.
• Audio input is not supported by the Converse API and is rejected with a clear error.

Native vLLM Backend¶

When using type: vllm, the router treats the backend as a vLLM OpenAI-compatible server:

• Default URL: None; the server URL must be specified (vLLM's own default is http://localhost:8000)
• Health Check: Uses /health as primary (available after model load), with /v1/models as fallback
• Model Discovery: Auto-discovers models from /v1/models; each entry's max_model_len feeds the model metadata context window when no explicit model-metadata.yaml limits are set. When several backends serve the same model ID with different max_model_len values, the router publishes the minimum (the window every backend can honor) and logs the disagreement at debug level; explicit model-metadata.yaml or model-metadata.d/ limits still take precedence
• Token Counting: /v1/messages/count_tokens tokenizes through the engine's /tokenize endpoint
• Passthrough: Non-standard sampling fields (top_k, min_p, repetition_penalty, ...) are forwarded unchanged
• Engine statistics: With engine_stats.enabled: true the router scrapes vLLM's GET /metrics (served on the API port, no extra server flag needed; --api-key does not guard it), plus /v1/models for context length and /version for the engine version

Minimal vLLM configuration:

backends:
  - name: "vllm-server"
    type: vllm
    url: "http://localhost:8000"
    # api_key only if the server was started with --api-key

Native SGLang Backend¶

When using type: sglang, the router provides native support for SGLang (the sglang.srt HTTP server, also the engine behind the SGLang Model Gateway). Supported aliases: SGLang, sg-lang, sg_lang.

• Default URL: http://localhost:30000 (sglang.srt default port)
• Health Check: Uses /health as primary, with /v1/models as fallback. SGLang's /health returns 503 while the engine is starting or exiting, which the router treats as warming up / unhealthy
• Model Discovery: Auto-discovers models from /v1/models; each entry's max_model_len (equal to the model's context length) feeds the model metadata context window when no explicit model-metadata.yaml limits are set. When several backends serve the same model ID with different max_model_len values, the router publishes the minimum (the window every backend can honor) and logs the disagreement at debug level; explicit model-metadata.yaml or model-metadata.d/ limits still take precedence
• Token Counting: /v1/messages/count_tokens tokenizes through the engine at /v1/tokenize with the vLLM-style {"model", "prompt"} payload
• Responses API: /v1/responses requests are converted to chat completions, like the rest of the vLLM family
• Passthrough: SGLang request extensions (top_k, min_p, min_tokens, regex, ebnf, repetition_penalty, lora_path, session_params, separate_reasoning, chat_template_kwargs, ...) are forwarded unchanged, and reasoning_content is preserved on messages and stream deltas, including the final usage chunk when stream_options.include_usage is set
• Authentication: Optional; when the server is started with --api-key, set api_key and the router sends Authorization: Bearer (SGLang keeps /health open even with a key)

Minimal SGLang configuration:

backends:
  - name: "sglang"
    type: sglang
    # No URL needed if using default http://localhost:30000
    # No API key required unless the server enforces one

Full SGLang configuration:

backends:
  - name: "sglang"
    type: sglang
    url: "http://192.168.1.100:30000"  # Custom URL if needed
    weight: 2
    api_key: "${SGLANG_API_KEY}"       # Optional: only if started with --api-key
    # Models are auto-discovered from /v1/models

Engine statistics: with engine_stats.enabled: true the router reads GET /v1/loads?include=core, which needs no extra server flag (a Bearer key is sent when the backend has one). Older servers without /v1/loads fall back to the deprecated GET /get_load, then to GET /metrics, which requires starting the engine with --enable-metrics. Data-parallel deployments report one entry per DP rank; the router sums counts and takes the maximum of usage fractions across ranks.

Native TensorRT-LLM Backend¶

When using type: tensorrt-llm, the router treats the backend as a TensorRT-LLM HTTP/OpenAI-compatible server. Supported aliases: TensorRTLLM, TensorRT-LLM, tensorrtllm, tensorrt_llm, trtllm, trt-llm, trt_llm.

• Default URL: None; the server URL must be specified
• Health Check: Uses /health as primary, with /v1/models as fallback
• Model Discovery: Auto-discovers models from /v1/models through the vLLM-compatible catalog parser
• Token Counting: /v1/messages/count_tokens tokenizes through the vLLM-compatible /tokenize path
• Responses API: /v1/responses requests are converted to chat completions, like the rest of the vLLM family
• KV listener salt echo: With prefix_routing.enabled: true and prefix_routing.salt_echo: true, the router injects its own cache_salt into chat-completion requests unless the client already supplied one. A continuum-kv-listener source configured with identity: salt_echo can read TensorRT-LLM blocks[].cache_salt events and attribute blocks without /detokenize. Malformed or backend-mismatched salts fall back to detokenization.

backends:
  - name: "trtllm"
    type: tensorrt-llm
    url: "http://localhost:8000"

Engine Tokenizer Endpoints¶

POST /tokenize and POST /detokenize (aliases /v1/tokenize and /v1/detokenize) reach the tokenizer of the engine that serves the requested model. Capability follows the backend type; there is no configuration field for it.

The four router paths are interchangeable on ingress; the path sent to the engine depends only on the backend type, as the table shows. The engine path is appended to backends[].url, and a base URL ending in /v1 (for example http://host:8000/v1) has that one trailing /v1 dropped first, so vllm, llamacpp, and mlxcel are still called at /tokenize and /detokenize and sglang at /v1/tokenize and /v1/detokenize. Other path prefixes, /openai included, are kept.

vLLM registers /tokenize and /detokenize unconditionally. This comes from reading vLLM main at commit af2a582d5c (register_vllm_serve_api_routers calls attach_tokenize_router with no task or flag condition), not from a live test. Only GET /tokenizer_info needs --enable-tokenizer-info-endpoint, and the router does not proxy it.

The llamacpp routes and response shapes, including the /detokenize answer {"content": "..."}, are verified against llama-server build 11344. The mlxcel row follows that project's route source, and the unsupported rows for tensorrt-llm, ollama, and lmstudio come from each project's source or API documentation; none of those four was verified against a live engine.

A model served only by unsupported types answers 501 tokenization_not_supported. When a model is served by both kinds, the request always goes to a supported backend. See Tokenize and Detokenize for the contract.

Native llama.cpp Backend¶

When using type: llamacpp, the router provides native support for llama.cpp llama-server:

• Default URL: http://localhost:8080 (llama-server default port)
• Health Check: Uses /health endpoint (with fallback to /v1/models)
• Model Discovery: Parses llama-server's hybrid /v1/models response format
• Rich Metadata: Extracts context window, parameter count, and model size from response
• Engine statistics: With engine_stats.enabled: true, the router reads GET /slots (enabled by default; --no-slots disables it) and, when started with --metrics, GET /metrics. The source is selected from endpoint_slots and endpoint_metrics in GET /props, where default_generation_settings.n_ctx supplies the per-slot context length (total context divided across parallel slots). If both endpoints are disabled, the router reports backend_engine_stats_scrape_success 0 (the same applies to type: mlxcel)

Minimal llama.cpp configuration:

backends:
  - name: "local-llama"
    type: llamacpp
    # No URL needed if using default http://localhost:8080
    # No API key required for local server

Full llama.cpp configuration:

backends:
  - name: "local-llama"
    type: llamacpp
    url: "http://192.168.1.100:8080"  # Custom URL if needed
    weight: 2
    # Models are auto-discovered from /v1/models endpoint

llama.cpp Features¶

Tool Calling Auto-Detection¶

The router automatically detects tool calling capability for llama.cpp backends by querying the /props endpoint during model discovery. This enables automatic function calling support without manual configuration.

How it works:

1. When a llama.cpp backend is discovered, the router fetches the /props endpoint
2. The chat_template field is analyzed using precise Jinja2 pattern matching to detect tool-related syntax
3. If tool calling patterns are detected, the model's function_calling capability is automatically enabled
4. Detection results are stored for reference (including a hash of the chat template)

Detection Patterns:

The router uses precise pattern matching to reduce false positives:

• Role-based patterns: message['role'] == 'tool', message.role == "tool"
• Tool iteration: for tool in tools, for function in functions
• Tool calls access: .tool_calls, ['tool_calls'], message.tool_call
• Jinja2 blocks with tool keywords: {% raw %}{% if tools %}, {% for tool_call in ... %}

Example /props response analyzed:

{% raw %}

{
  "chat_template": "{% for message in messages %}{% if message['role'] == 'tool' %}...",
  "default_generation_settings": { ... },
  "total_slots": 1
}

Fallback Behavior:

• If /props is unavailable: Tool calling is assumed to be supported (optimistic fallback for modern llama.cpp versions)
• If /props returns an error: Tool calling is assumed to be supported (ensures compatibility with newer models)
• If chat template exceeds 64KB: Detection is skipped and defaults to supported
• Detection is case-insensitive for maximum compatibility
• Results are merged with any existing model metadata from model-metadata.yaml
• Detected capabilities appear in the features field of the /v1/models/{model_id} response

Model Metadata Extraction¶

The router extracts rich metadata from llama-server responses:

Starting llama-server¶

# Basic startup
./llama-server -m model.gguf --port 8080

# With GPU layers
./llama-server -m model.gguf --port 8080 -ngl 35

# With custom context size
./llama-server -m model.gguf --port 8080 --ctx-size 8192

Auto-Detection of llama.cpp Backends¶

When a backend is added without a type specified (defaults to generic), the router automatically probes the /v1/models endpoint to detect the backend type. llama.cpp backends are identified by:

1. owned_by: "llamacpp" in the response
2. Presence of llama.cpp-specific metadata fields (n_ctx_train, n_params, vocab_type)
3. Hybrid response format with both models[] and data[] arrays

This auto-detection works for:

• Hot-reload configuration changes
• Backends added via Admin API without explicit type
• Configuration files with type: generic or no type specified

Example: Auto-detected backend via Admin API:

# Add backend without specifying type - auto-detects llama.cpp
curl -X POST http://localhost:8080/admin/backends \
  -H "Content-Type: application/json" \
  -d '{
    "name": "local-llm",
    "url": "http://localhost:8080"
  }'

Native MLxcel Backend¶

When using type: mlxcel, the router provides native support for MLxcel, an MLX-based model serving backend for macOS with Apple Silicon:

• Default URL: http://localhost:8080 (same as llama-server)
• API Compatibility: Fully compatible with llama-server (llama.cpp) API
• Model Format: Serves SafeTensor format models via Apple's MLX framework
• Health Check: Uses /health as primary, with /v1/models as fallback
• Platform: macOS with Apple Silicon only

Minimal MLxcel configuration:

backends:
  - name: "mlxcel-local"
    type: mlxcel
    # No URL needed if using default http://localhost:8080

Full MLxcel configuration:

backends:
  - name: "mlxcel-local"
    type: mlxcel
    url: "http://192.168.1.100:8080"  # Custom URL if needed
    weight: 2
    models:
        - mlx-community/Qwen3-4B-4bit

Native LM Studio Backend¶

When using type: lmstudio, the router provides native support for LM Studio local server:

• Default URL: http://localhost:1234 (LM Studio default port)
• Health Check: Uses /v1/models (OpenAI-compatible) as primary, with /api/v1/models (native API) as fallback
• Model Discovery: Auto-discovers models from /v1/models endpoint
• owned_by Attribution: Reports "lmstudio" for proper model attribution
• Engine statistics: With engine_stats.enabled: true the router reads GET /api/v0/models, which needs no extra server flag; only loaded models (state: "loaded") enter the snapshot, with max_context_length as the reported context length. LM Studio has no running/waiting request count, so those series stay absent by design, not as a scrape failure

Minimal LM Studio configuration:

backends:
  - name: "lmstudio"
    type: lmstudio
    # No URL needed if using default http://localhost:1234
    # No API key required for local server

Full LM Studio configuration:

backends:
  - name: "lmstudio"
    type: lmstudio
    url: "http://192.168.1.100:1234"  # Custom URL if needed
    weight: 2
    api_key: "${LM_API_TOKEN}"        # Optional: LM Studio API token (v0.4.0+)
    # Models are auto-discovered from /v1/models endpoint

LM Studio Features¶

Native Continuum Router / Backend.AI GO Backend¶

When using type: continuum-router, the router connects to a remote Continuum Router instance or Backend.AI GO deployment for federated LLM routing. Supported aliases include: continuum-router, continuum_router, ContinuumRouter, backendai, backend-ai, backend_ai.

• Health Check: Uses /health as primary, with /v1/models as fallback
• Model Discovery: Auto-discovers models from the remote instance's /v1/models endpoint
• Authentication: Bearer token via Authorization: Bearer <key> header
• Request Passthrough: Requests are forwarded with no transformation (both systems use OpenAI-compatible APIs)
• owned_by Attribution: Reports "continuum-router" for discovered models
• Transport: Supports both HTTP and Unix Domain Socket transports

Minimal configuration:

backends:
  - name: "remote-cr"
    type: continuum-router
    url: "https://remote.example.com"
    api_key: "${REMOTE_API_KEY}"
    # Models are auto-discovered from remote /v1/models endpoint

Full configuration with explicit models:

backends:
  - name: "remote-backendai"
    type: continuum-router
    url: "https://remote-backend-ai.example.com"
    api_key: "${REMOTE_BACKEND_AI_API_KEY}"
    weight: 2
    models:
      - gpt-5.6-sol
      - claude-sonnet-4-5-20250929

Use cases:

• Multi-region deployment: geo-route requests across Continuum Router instances
• Federated routing: connect multiple independent CR or Backend.AI GO deployments
• Tiered access: route through a central Backend.AI GO instance for quota management
• High availability: configure multiple Backend.AI GO instances for failover

Continuum Router Backend Features¶

Forwarding Loops and the Hop Limit¶

A backend that points at a router (including this one) turns each request it serves into a nested request on that router. A backend whose URL is the router's own listener, or a cycle of routers (A -> B -> A), would otherwise nest without bound while each hop holds its inbound request open waiting on the next. This applies to any backend type (continuumrouter, generic, openai, or a Hub-managed backend topology) that targets a router.

Every router tracks the number of routers a request has traversed in the x-continuum-router-hops header:

• Pre-admission check: Every request is inspected before authentication, rate limiting, and the concurrency ceiling so none of those can reject a looping request first and trigger an upstream retry:
  • An absent header counts as 0.
  • A request arriving with a count of 8 or more is rejected with 508 Loop Detected (type: server_error, code: router_loop_detected) before any handler starts, returning a JSON error even when stream: true. On /anthropic/* routes the body uses the Anthropic error envelope with type: api_error.
  • Otherwise, the header is rewritten to a single value equal to the inbound count plus 1.
• Outbound propagation:
  • Every dispatch that forwards client headers sends the incremented count regardless of backend type.
  • Paths that build their own headers set it explicitly: tokenizer routes, /anthropic/v1/messages/count_tokens, image edits, the realtime WebSocket handshake, the /v1/decisions passthrough, and control-plane provider batch calls (/v1/batches create, retrieve, list, cancel, and results; the background batch poller serves no request and sends none).
• Internal sub-requests (guardrails and System One):
  • Calls made on a request's behalf carry that request's hop count: LLM-transport guardrails (self_hosted_classifier, classifier, custom_classifier, by endpoint or backend; webhook, by endpoint) and System One decision calls (systemone guardrail, smart-routing system_one classifier, safeguards.action_safety with evaluator: systemone, and fallback.quality_check).
  • Self-referential guardrail or /v1/systemone emulation targets therefore terminate at the hop limit. A guardrail endpoint's 508 is not retried, does not trip the guardrail's circuit breaker, and fails closed regardless of on_error.
• Post-reply quality judgment: usage.quality_judgment runs after the reply and sends the hop count of the judged request. A judgment target that loops back through /v1/systemone adds at least two hops per cycle, capping a single client request at four judgments before terminating with 508 (recorded as failed).
• /v1/decisions passthrough: A 508 from a /v1/decisions target is returned as 508 router_loop_detected without advancing fallback_chains.
• Header validation and reservation:
  • Any value that is not a non-negative integer (including an empty value) is rejected with 400 invalid_router_hops. When multiple values are present (separate header lines or comma-separated), the largest value is used; values exceeding the 32-bit range are rejected as a loop.
  • The hop limit of 8 is fixed (legitimate chains are typically two or three routers deep).
  • x-continuum-router-hops is reserved: request_extensions.headers cannot set it in any casing.

A loop terminates after at most nine nested requests, and the innermost 508 propagates outward as the final response: it is never retried, never advances a fallback chain, and never counts against the backend's circuit breaker (whose default failure_status_codes are 500, 502, 503, and 504).

At startup and during config validate, the router warns when a backend URL's host:port matches a server.bind_address entry (0.0.0.0 and [::] match any loopback host on the same port, and localhost matches loopback addresses). Because this check does not resolve DNS names, it catches direct self-references while the runtime hop limit catches multi-node cycles. /v1/models aggregation builds its own requests and does not send the hop header.

Loop refusals are rate-limited in the logs: the first refusal logs at WARN, repeats within the same minute log at DEBUG, and a summary with the suppressed count is logged when the window closes (by the next refusal, or by a background flush within 30 seconds when none arrives).

Unix Domain Socket Backends¶

Continuum Router supports Unix Domain Sockets (UDS) as an alternative transport to TCP for local LLM backends. Unix sockets provide:

• Enhanced Security: No TCP port exposure - communication happens through the file system
• Lower Latency: No network stack overhead for local communication
• Better Performance: Reduced context switching and memory copies
• Simple Access Control: Uses standard Unix file permissions (on Linux/macOS; Windows does not support Unix file modes)

URL Format:

unix:///path/to/socket.sock

Platform Support:

Configuration Examples:

On Windows, use drive-letter paths (e.g., unix://C:/temp/llama.sock). On Linux/macOS, use standard absolute paths (e.g., unix:///var/run/llama.sock).

backends:
  # llama-server with Unix socket (Linux/macOS)
  - name: "llama-socket"
    type: llamacpp
    url: "unix:///var/run/llama-server.sock"
    weight: 2
    models:
      - llama-3.2-3b
      - qwen3-4b

  # Ollama with Unix socket
  - name: "ollama-socket"
    type: ollama
    url: "unix:///var/run/ollama.sock"
    weight: 1
    models:
      - llama3.2
      - mistral

  # vLLM with Unix socket
  - name: "vllm-socket"
    type: vllm
    url: "unix:///tmp/vllm.sock"
    weight: 3
    models:
      - meta-llama/Llama-3.1-8B-Instruct

Starting Backends with Unix Sockets: