Error Handling¶

Continuum Router returns OpenAI-compatible JSON errors at the HTTP boundary and uses retries, active health filtering, and model fallback to contain transient backend failures.

Error response format¶

Most router-generated failures use this shape:

{
  "error": {
    "message": "The service is temporarily unavailable. Please try again later.",
    "type": "service_unavailable",
    "param": null,
    "code": "service_unavailable"
  }
}

All four members are always present, matching the OpenAI Error schema: message and type are strings, and param and code are a string or null rather than being omitted.

code is a stable machine-readable identifier rather than the HTTP status code (which is carried on the HTTP status line). param names the offending request field when the upstream provider supplies one. A rate-limited response may also include Retry-After.

When an upstream provider supplies its own string code, that code passes through unchanged, so a client can still branch on values such as insufficient_quota or context_length_exceeded. The table below lists the codes the router itself authors.

Upstream output-token errors¶

When a backend attributes a rejected output budget to a known field, the router preserves the client's field name: max_tokens, max_completion_tokens, or Responses max_output_tokens. Errors containing engine limit identifiers such as max_model_len or max_total_tokens receive a client-facing explanation that asks the caller to adjust the budget or shorten the input. An engine configuration value is not advertised as the usable output budget.

Other backend errors retain their sanitized diagnostic text. Errors without a known offending field keep param: null. Anthropic errors use the same message rule in their own envelope; original upstream diagnostics remain available in server logs.

Router-authored error codes¶

Streaming failures that happen before any bytes are sent use an HTTP error response. Once an SSE response has started, the stream reports an error event and terminates; the status code can no longer be changed. On the Chat Completions wire the error event carries the same error object as the HTTP body. On the Responses wire it is the flat ResponseErrorEvent, with code, message, param, and sequence_number beside type.

Status codes¶

Backend error messages¶

When a backend rejects a request with 4xx, the router relays the backend's own message so the client can act on it. For example, a context-length overflow and an invalid parameter return different messages. /v1/chat/completions and /anthropic/v1/messages return the same message, each in its own error format. On /v1/responses, a JSON error body from a backend that serves the Responses API natively is forwarded unchanged, and the converted paths follow the rules below.

• Recognized body shapes: OpenAI and Gemini {"error":{"message":...}}, Anthropic {"type":"error","error":{...}}, {"error":"..."}, a top-level message, FastAPI detail (string or validation list), and short plain text.
• Redaction: Absolute URLs, inline credentials, Bearer tokens, provider API keys, key=value secrets, IP addresses, and host:port endpoints are replaced by [redacted-url], [redacted], or [redacted-host]. Control characters are removed, and the message is limited to 500 bytes.
• Anthropic error type: On /anthropic/v1/messages, a backend 400 or 422 becomes invalid_request_error, 413 becomes request_too_large, and 529 becomes overloaded_error.
• Unrecognized 4xx body: The body is relayed after Backend rejected the request: HTTP <status>:, redacted and bounded, because it is usually the only description of what the backend rejected.
• Generic fallback: A 5xx body, an HTML page, an empty body, or a binary body gets a generic per-status message instead, so stack traces and infrastructure details do not reach the client.

The operator log always records the full backend body.

Retries¶

The top-level retry section controls retries to the selected backend:

retry:
  max_attempts: 3
  initial_delay: 100ms
  max_delay: 10s
  backoff_multiplier: 2.0
  jitter: true
  retryable_status_codes: [429, 502, 503, 504]
  retryable_errors: [ConnectionError, TimeoutError]
  timeout: 30s

These are also the default values. The router applies exponential backoff with optional jitter. A provider 429 is retried only when it is classified as a transient rate limit; quota, credit, and billing exhaustion fail fast. Upstream Retry-After or Google retry-delay hints are honored after validation and are capped at 24 hours.

Retries repeat an attempt against a backend. They are distinct from fallback, which can select another model or provider after an eligible failure.

Circuit breaker¶

Continuum Router includes a configurable three-state circuit-breaker implementation and Admin controls:

circuit_breaker:
  enabled: true
  failure_threshold: 5
  failure_rate_threshold: 0.5
  minimum_requests: 10
  timeout: 60s
  half_open_max_requests: 3
  half_open_success_threshold: 2
  failure_status_codes: [500, 502, 503, 504]
  timeout_as_failure: true
  sliding_window_size: 60s
  sliding_window_type: count_based
  backends:
    openai:
      failure_threshold: 10
      timeout: 120s

sliding_window_type accepts count_based or time_based. Entries below circuit_breaker.backends override only the supported per-backend fields.

The regular LLM proxy path records request outcomes in this state machine and consults it during backend selection: an open backend circuit is excluded from selection, and when every compatible circuit is open the router returns a 503 service-unavailable error and participates in configured fallback. It complements active health checks, retries, and fallback rather than replacing them. See Circuit Breaker for the outcome-recording and admission details.

When Admin API authentication permits access, operators can inspect or control breaker state through:

See Admin API for authentication and complete endpoint details.

Model fallback¶

Fallback chains choose alternative models after configured failure conditions:

fallback:
  enabled: true
  mid_stream_enabled: true
  fallback_chains:
    gpt-5.4:
      - gpt-5.4-mini
      - claude-sonnet-4-6
  fallback_policy:
    trigger_conditions:
      error_codes: [429, 500, 502, 503, 504]
      timeout: true
      connection_error: true
      model_not_found: true
      backend_unhealthy: true
    max_fallback_attempts: 3
    fallback_timeout_multiplier: 1.5
    preserve_parameters: true
  model_settings:
    gpt-5.4:
      fallback_enabled: true
      notify_on_fallback: true

With notify_on_fallback: true (the default), successful fallback responses can contain:

• X-Fallback-Used
• X-Original-Model
• X-Fallback-Model
• X-Fallback-Reason
• X-Fallback-Attempts: total model attempts including the primary; always at least 2 when a fallback served the response

Set notify_on_fallback: false for a model to suppress these headers while still serving the fallback.

Per-request opt-out¶

A /v1/chat/completions or /v1/completions request body can carry "disable_fallbacks": true to skip every cross-model hop for that request only, before the first byte and mid-stream alike. The primary model's error is returned instead, while retries and failover between backends that serve the same model still apply and other requests keep their chains. The router removes the field before dispatch, whatever its value, so a backend never receives it. A Continuum Hub-attributed request may set it too, since it only narrows what the router does for that request.

Ingress coverage and error format¶

• Covered routes: Fallback chains run on every ingress that resolves a backend from the requested model. OpenAI-shaped routes use the shared dispatch funnel, while /anthropic/v1/messages and /v1/responses run their own per-attempt selection and native dispatch on both non-streaming requests and streaming requests before the first SSE byte.
• Streaming handshakes: A streaming attempt whose upstream handshake succeeded is committed and does not hop.
• Excluded routes: /v1/responses/compact and count_tokens do not run a fallback chain.
• Cross-provider wire conversion: Cross-provider fallback keeps the request payload in canonical form and swaps only the model name; wire-format conversion for the newly selected backend happens at dispatch rather than at the hop.
• Exhausted chain or non-trigger failure: When the chain is exhausted or the failure is not a configured trigger, the router returns the last attempt's error in that ingress's dialect ({"type":"error",...} on /anthropic/v1/messages, {"error":{...}} on /v1/responses). See Model fallback for per-ingress rules.

Timeout scaling and mid-stream recovery¶

• fallback_timeout_multiplier (1.0–5.0, default 1.5): Scales each hop's request timeout. The primary attempt uses its configured base timeout, and hop n runs with base * multiplier^(n-1), clamped to the timeouts.limits ceiling for that timeout class. Because base is the timeout that attempt would have used without a chain, a longer per-model budget remains proportionally longer on the hop.
• Unscaled timeouts: fallback_timeout_multiplier does not scale timeouts.connection, the wait for a fallback dial permit, within-hop retries, or a mid-stream hop after the client already holds the stream.
• Composition with timeouts.streaming_fallback_budget_multiplier: timeouts.streaming_fallback_budget_multiplier sizes the single wall-clock budget shared across all attempts of a streaming request. The two settings compose: a streaming hop is scaled by fallback_timeout_multiplier first and then capped at the remaining shared wall-clock budget.
• mid_stream_enabled: Setting mid_stream_enabled: true permits mid-stream recovery on streaming paths that support it. Because mid-stream recovery may buffer output, disable it when immediate token delivery is more important than recovery after a stream has begun.

Timeouts and request size¶

Configure router and model deadlines in the timeouts section and backend request limits in the documented server/backend sections. Avoid making retry timeouts longer than the caller's total deadline; the caller can otherwise disconnect while retry work continues.

Use the generated configuration as the canonical field reference:

continuum-router --generate-config > config.yaml
continuum-router config validate config.yaml

Observability and diagnosis¶

Use implemented interfaces rather than undocumented debug endpoints:

# Public process/backend health
curl http://localhost:8080/health

# Admin health and circuit state (add configured admin credentials)
curl http://localhost:8080/admin/health
curl http://localhost:8080/admin/circuit/all

# Validate configuration without starting the server
continuum-router config validate /etc/continuum-router/config.yaml

Prometheus metrics are exposed at the configured metrics path when the metrics feature and runtime metrics configuration are enabled. Logs use tracing; select verbosity with logging.level, CONTINUUM_LOG_LEVEL, or RUST_LOG as appropriate.

Backend refusal log rate limiting¶

When no backend can serve a model (every backend is unhealthy, or every backend has reached its per-backend rate limit), repeated refusal logs are rate-limited per log site and model:

• First refusal: Written at WARN with full detail.
• Within the 60-second window: Subsequent refusals for the same (site, model) within 60 seconds are written at DEBUG only, so a client retrying in a tight loop adds at most one WARN line per site and model per minute. Set RUST_LOG=info,continuum_router=debug to inspect every individual refusal.
• Post-window summary: When the 60-second window closes with suppressed repeats, a single WARN summary is written (for example, 500 requests refused for model 'gpt-4o' in the last 61s: all backends unhealthy) carrying structured site, model, reason, refused, and window_secs fields. The first refusal after the window writes it, and a background flush that runs every 30 seconds writes it when no further refusal arrives, so a burst that stops mid-window still reports its final count, at most 30 seconds after the window closes. Whichever runs first writes the summary, so it is never written twice. A window without suppressed repeats writes no summary, and an idle router writes nothing. Prometheus metrics count every refused request either way.
• Alerting level (WARN vs. ERROR): Refusal lines are logged at WARN, not ERROR. Alerts matching Router error: All backends are unhealthy, Exhausted all retry attempts, or No healthy backend found should target the WARN level or alert on the refusal metrics instead.

The router does not provide /admin/errors/* or /admin/debug/error endpoints. Inspect structured logs, metrics, /admin/health, and /admin/circuit/* instead.

Client recommendations¶

1. Set a caller-side deadline longer than the expected router/backend timeout.
2. Retry only idempotent operations unless your application has deduplication.
3. Use exponential backoff with jitter and honor Retry-After.
4. Do not retry authentication, permission, validation, or quota-exhaustion errors without changing the request or credentials.
5. Record a correlation/request identifier from your own client logs and avoid logging API keys or resolved configuration output.