Model Fallback¶

Model fallback maps a requested model to an ordered list of alternatives. It can retry through that list after configured HTTP errors, timeouts, connection failures, model-not-found results, or unhealthy-backend selection failures.

Configuration¶

fallback:
  enabled: true
  mid_stream_enabled: true

  fallback_chains:
    gpt-5.4:
      - gpt-5.4-mini
      - claude-sonnet-4-6

  fallback_policy:
    trigger_conditions:
      error_codes: [429, 500, 502, 503, 504]
      timeout: true
      connection_error: true
      model_not_found: true
      backend_unhealthy: true
    max_fallback_attempts: 3
    fallback_timeout_multiplier: 1.5
    preserve_parameters: true
    max_concurrent_dials_per_backend: 50

  model_settings:
    gpt-5.4:
      fallback_enabled: true
      notify_on_fallback: true

fallback_chains keys are primary models and values are tried in order. Chains cannot be empty, self-referential, or cyclic. Model names are limited to 128 alphanumeric, ., _, or - characters.

fallback_policy.max_fallback_attempts is 1–10. fallback_timeout_multiplier is 1.0–5.0: it scales each hop's own request timeout. The primary is attempt 1 and keeps the timeout the configuration resolved for it, and hop n runs with base * multiplier^(n-1), clamped to the timeouts.limits ceiling for that timeout class, so a chain cannot reach past a value the configuration itself could not have set. The base is whatever the attempt would have used on its own: the per-model timeouts.request.standard.total on the OpenAI-compatible non-streaming path, the per-attempt streaming window on the pre-stream connection phase, and the per-model profile (total, plus the first-SSE-byte deadline on the streaming arm) on /anthropic/v1/messages and /v1/responses. It does not scale timeouts.connection, the wait for a fallback dial permit, a within-hop retry, or a mid-stream hop taken after the client already holds the stream. It is not timeouts.streaming_fallback_budget_multiplier, which sizes the one wall-clock budget shared by every attempt of a streaming request; the two compose rather than multiply, because the connection phase scales the hop's window first and then caps it at what is left of that budget. max_concurrent_dials_per_backend is 0–10000 (0 = unlimited, default 50): the per-backend ceiling on concurrent fallback hop dials, shared by every fallback path and held only until the provider handshake returns. See Concurrent request storms for the hold scope, the saturation behavior, and the reload semantics.

The schema also accepts trigger_conditions.circuit_breaker_open. The proxy now consults circuit-breaker state during selection, but a request rejected by an open circuit is surfaced as a backend_unhealthy trigger, not as the distinct circuit_breaker_open reason, so that specific condition is not emitted by ordinary LLM traffic. Enable backend_unhealthy (and active health checks) to advance the fallback chain when a backend's circuit is open or the backend is unhealthy.

Pre-stream execution¶

For a non-streaming request or a stream that has not begun, the router:

1. resolves the primary model and its configured chain;
2. executes the primary attempt;
3. classifies an eligible failure;
4. selects the next model and rewrites the payload's model name, which is the only thing the hop changes;
5. stops at the first success or at max_fallback_attempts.

Fallback is model-based. Each alternative model is resolved through the normal backend pool, health filtering, access policy, and selection strategy.

Cross-provider translation¶

A fallback hop payload stays in canonical OpenAI chat-completions form from end to end, and wire-format conversion happens exactly once, at dispatch, keyed on the selected backend's configured backend_type. Messages, tools, tool_choice, stop, and every sampling parameter therefore reach the backend transform exactly as the client sent them, and the transform, which is the only layer that knows which backend actually serves the hop, produces the provider-native request from that canonical input.

The hop itself performs exactly one thing: it swaps the payload's model name for the hop target. Nothing is added, removed, or reshaped, and no parameter is pre-filtered, because the hop can only infer a provider from the model name and that inference is wrong for claude-* models served through Bedrock, Gemini behind its OpenAI-compatible endpoint, gpt-prefixed open models on vLLM, and every operator-chosen alias, which infers no provider at all.

Three Anthropic-native fields ride on a canonical body and are not part of the OpenAI chat-completions schema: speed (fast mode) and the extended-thinking pair thinking and output_config. They are removed at dispatch, keyed on the selected backend's configured backend_type, not at the hop:

• Every OpenAI-wire type (generic, openai, azure, gemini, vllm, ollama, llamacpp, mlxcel, lmstudio, sglang) has them stripped from a /v1/chat/completions body at every send site. No OpenAI-wire target reads them under these names, and a strict cloud endpoint answers HTTP 400 for an unrecognized top-level key, which would kill the very hop that exists to rescue the request.
• anthropic and bedrock keep them: their transform is the code that reads thinking and output_config, and speed stays governed by the backend's separate anthropic_fast_mode opt-in.
• continuumrouter keeps them as well. A downstream Continuum Router understands the same canonical extension fields and applies this same rule against its own backends, so stripping here would silently discard the client's reasoning intent on a hop toward a downstream router that fronts Anthropic.
• The rule is left untouched only when the selected backend's name is absent from the configuration snapshot dispatch consults, for example a hot reload that removed or renamed the backend while the request was in flight; guessing there would be the same inference this design removes. A backend declared without type: defaults to generic, which strips like every other OpenAI-wire type, so omitting type: does not preserve the fields.

reasoning_effort is never stripped, and a value the client sent is never overwritten. It is the provider-neutral spelling every target either honors or ignores harmlessly, so it is what carries a client's reasoning intent across a hop, which is also why the conversion described next writes into that key when the client did not set it. extra_body is never touched either: stripping is top-level only.

Before v1.27.0 the removal lived at the hop, keyed on the providers inferred from the two model names. That table could not express the rule. Two custom aliases both inferred unknown, which it read as "same provider, nothing to do", and all three fields reached an OpenAI wire; a claude-* primary hopping onto an Anthropic-typed backend behind a custom alias looked like a provider change, and the client's thinking was dropped by the one layer that had no way to know the target would have honored it.

The same release removed the hop's pre-conversion toward the target's native shapes, which collided with the conversion dispatch performs: pre-translated tools were skipped by the second conversion and arrived empty, a pre-translated tool_choice was rejected outright and failed the whole hop, and a stop renamed to stop_sequences was never read. Those conversions were removed.

The thinking config becomes reasoning_effort¶

Stripping alone is fail-safe but lossy: a request that asked for extended thinking would hop onto a target running with its own default reasoning behavior. From v1.27.0 the same dispatch-side gate translates before it strips. On exactly the requests that strip, the Anthropic thinking config is mapped to a reasoning level and written as reasoning_effort, and only then are the three fields removed. The mapping is the inverse of the OpenAI-to-Anthropic direction and reuses its tables, so the two stay inverses of each other.

The budget rows are not specific to a hop. The /anthropic/v1/messages ingress resolves budget_tokens through the same function, ReasoningEffort::from_budget_tokens, on both its Chat Completions transform and the Responses API transform it uses for responses_only models, so a budget resolves to the same reasoning level whichever door the request enters through.

output_config.effort wins over thinking, except for the disabled case where thinking wins because there is no effort to express, and an unrecognized effort string falls through to the budget. none, minimal, auto, and max are never emitted: some target rejects each of them, and the ones that mean "use the default" are expressed by emitting nothing at all.

An existing reasoning_effort, or its nested reasoning.effort spelling, always wins and is never overwritten. It is the client's explicit, provider-neutral statement of intent, and the Anthropic-native fields are the derived form of the same thing. A malformed thinking config degrades to a plain strip, never to a failed hop.

Fitting the level to the target¶

The conversion runs at the send site, after the reasoning validation that guards a client-supplied reasoning_effort, so a fabricated value would otherwise reach the wire unchecked. It is fitted to what the selected target documents:

• openai and azure consult the OpenAI model table. A model with no reasoning_effort support at all (gpt-4o, the chat and instant tiers, and any Azure deployment name the table does not recognize) receives nothing, and the hop proceeds as a plain strip. Otherwise the level is fitted to that model's documented set, because xhigh is not the only possible mismatch: gpt-5-pro allows high only, the legacy pro tiers allow medium, high, and xhigh, and the o-series and codex tiers allow low, medium, and high. An allowed level is used as is; xhigh toward a target without it becomes high; otherwise the closest allowed level on the low < medium < high < xhigh ladder wins, preferring the next higher level on a tie, so low toward gpt-5-pro becomes high and low toward gpt-5.2-pro becomes medium. A reasoning-capable GPT-5 variant the table does not describe yet is treated as allowing low, medium, and high. For azure, the deployment name in model is trusted to name the family it fronts, the same way the validator already trusts it for a client-supplied reasoning_effort, so a deployment named after a family it does not actually serve can now be fitted a level that family rejects; name Azure deployments after the model they serve.
• Every other OpenAI-wire target (generic, gemini, and the self-hosted engines vllm, ollama, llamacpp, mlxcel, lmstudio, sglang) receives low, medium, or high, never xhigh.

Local-engine targets receive the derived effort¶

Self-hosted engines are included deliberately. reasoning_effort is the canonical chat-completions spelling the router already treats as universally accepted: it is kept out of every strip precisely because every target either consumes it or ignores it, and the inbound /v1/messages handler has fabricated it toward these same engines since it shipped. vLLM, SGLang, llama.cpp, Ollama, and LM Studio accept the field on their OpenAI-compatible surfaces or ignore an unrecognized sampling knob rather than failing the request, and cloud Gemini's own transform maps it to thinking_level, so a Gemini target benefits directly. Withholding it would silently drop the reasoning intent for exactly the self-hosted targets a rescue hop most often lands on.

Streaming fallback¶

fallback.mid_stream_enabled controls whether the router buffers an active SSE response so it can recover after an upstream stream failure:

• true (default): mid-stream fallback and its per-stream buffer are allowed.
• false: pre-stream fallback remains available, but a failure after SSE output begins is returned as a stream error.

streaming.mid_stream_fallback.enabled is a mode selector, not the buffering switch:

• true: build a continuation request from accumulated output when enough content exists.
• false: restart the original request on the fallback model.

Configure streaming.mid_stream_fallback.max_fallback_attempts, min_accumulated_tokens, fallback_delay_ms, and continuation_prompt to tune this path. The stream's total and chunk-interval deadlines still apply.

Cross-provider streaming hops¶

A streaming fallback hop re-enters the same typed per-backend dispatch direct routing uses, keyed on the selected backend's configured backend_type. A hop onto a native Anthropic, Bedrock, or Gemini backend is served through that provider's native streaming pipeline (for Anthropic, the Messages API at /v1/messages with x-api-key and anthropic-version), and the client keeps receiving OpenAI-format SSE. Configuring a chain for a model whose primary backend is native does not affect healthy direct streaming to it.

The primary attempt takes the same dispatch, so the walk does not depend on the primary's wire either. A native-typed primary (anthropic, gemini, or a bedrock Runtime or Converse endpoint) that fails before the provider answers, which is a refused connect, a transport error, a handshake timeout, or a circuit rejection, enters the same pre-stream chain walk an OpenAI-wire primary enters, in both mid_stream_enabled modes and under the same trigger conditions and attempt limits, and the rescue's response carries the same X-Fallback-* headers. A chain entry that resolves to a native backend is not terminal either: a dead native entry advances to the next entry the way a dead OpenAI-wire entry does. An answered Gemini or Bedrock status follows the OpenAI-wire status policy: a non-429 4xx response is terminal, while 429 and 5xx responses advance only when their exact status appears in fallback.fallback_policy.trigger_conditions.error_codes. Refused connections remain governed by trigger_conditions.connection_error. Anthropic-shape native responses retain their existing behavior, and a native primary reached over a Unix socket is served as a whole, so a failure there is the client's answer.

The recovery point decides what a hop can reach:

• Before the SSE response is committed, which covers the initial connection and every pre-stream hop in both mid_stream_enabled modes, a hop can land on any backend type, and the attempt that fails can be on any backend type too. The response is committed at the first successful provider handshake, so pre-stream exhaustion is returned as a proper HTTP error rather than a 200 stream carrying an error event, and the X-Fallback-* headers below can still be attached.
• After SSE output has begun, mid-stream recovery speaks only the OpenAI-compatible wire and the response headers are already sent, so a chain entry that resolves to a native-protocol or Unix-socket backend is skipped deterministically and the next entry is tried; it is never turned into a request-wide error. A stream served through a native pipeline likewise has no mid-stream recovery of its own: a failure after its output begins surfaces as a normal stream error.

The same split decides which tuning knobs apply. A pre-stream hop is governed by fallback.fallback_policy.trigger_conditions and fallback.fallback_policy.max_fallback_attempts, the same gates non-streaming fallback uses, while streaming.mid_stream_fallback.max_fallback_attempts bounds only the hops the committed relay performs. Because the response is committed at the first successful handshake rather than up front, no SSE keep-alive comments flow while the router is still connecting, matching the mid_stream_enabled: false path and a request with no chain at all. That silent window is bounded: the connection phase shares one cross-attempt budget with the relay (timeouts.request.streaming.total scaled by timeouts.streaming_fallback_budget_multiplier), caps each handshake attempt, on either wire, at what is left of it, and answers with the preserved upstream failure once it is gone rather than dialing the rest of the chain at full length. A handshake the phase does obtain is always served, with its read window measured from the moment the relay takes it over.

Ingress coverage¶

A chain is keyed on the model a request dispatches on, and it fires on every ingress that resolves a backend from that name. The OpenAI-shaped ingresses (/v1/chat/completions, /v1/completions, /v1/embeddings, rerank, sparse embeddings, and /v1/images/generations) run it through the shared dispatch funnel. The two provider-shaped ingresses, /anthropic/v1/messages and /v1/responses, dispatch natively and run the same chain through their own per-attempt selection and dispatch, so the trigger conditions, max_fallback_attempts, and the per-backend hop dial bound are the ones chat applies. /v1/responses/compact and count_tokens do not run a chain.

On the two provider-shaped ingresses each attempt keeps its wire fidelity: the primary attempt is the native Anthropic Messages or Responses request as the client sent it, and a hop is the same typed request with only the model name swapped, converted at dispatch by the selected backend's backend_type exactly as a direct request to that model would be. On /v1/responses this holds for the passthrough strategy and for every convert strategy. An attempt is judged from what it returned: a 2xx commits it, a refused connect, a timeout, an unknown model, or no admissible backend hops under connection_error, timeout, model_not_found, or backend_unhealthy, and a provider-returned status hops only when its exact code appears in trigger_conditions.error_codes. Two failures are classified by status rather than class on these ingresses: a router-side total-timeout answer (504) and a convert-strategy connect failure (502), both of which sit in the default error_codes. When the chain is exhausted, or when the first failure does not trigger (a 400, for example), the client receives the last attempt's own error in the ingress's own dialect; nothing router-authored replaces a provider answer.

The streaming arms of both ingresses hop before the first byte only. A failure before the provider handshake selects the next chain entry and restarts the native stream on it, the response is committed at the first successful handshake, and the X-Fallback-* headers below ride on that committed response. Once the handshake is back no hop follows: a body that fails after that point is the client's answer, and neither ingress has mid-stream recovery, which remains an OpenAI-compatible-wire feature of /v1/chat/completions as described above.

Response headers¶

When a model's notify_on_fallback is true (the default) and a fallback succeeds, the response can contain:

X-Fallback-Used: true
X-Original-Model: gpt-5.4
X-Fallback-Model: gpt-5.4-mini
X-Fallback-Reason: error_code_429
X-Fallback-Attempts: 2

The router emits five fallback headers: X-Fallback-Used, X-Original-Model, X-Fallback-Model, X-Fallback-Reason, and X-Fallback-Attempts. Setting a model's notify_on_fallback: false suppresses all of them while still serving the fallback response.

X-Original-Model names the model the client asked for. When the request named a metadata alias that the router dispatched under its canonical id, the header carries the alias the client sent, not the canonical id the chain was keyed on; X-Fallback-Model still names the model that actually served.

X-Fallback-Attempts reports the total number of model attempts made to satisfy the request, counting the primary attempt. Because the header appears only when a fallback backend produced the response, its value is always at least 2: a value of 2 means the primary model failed once and the first fallback succeeded, and larger values indicate additional failed hops before success. This lets a client distinguish a clean first-hop recovery from a chain that traversed several failing models.

These headers are attached to non-streaming responses, and to streaming responses whose fallback happened before any SSE output: a selection-time walk onto a fallback model, or a pre-stream chain advance, in both mid_stream_enabled modes. They are emitted on every ingress that runs a chain, including /anthropic/v1/messages and /v1/responses on both their non-streaming and pre-stream streaming arms. A streaming (SSE) response sends its headers before the body, so a mid-stream recovery cannot report the attempt count in a header; mid-stream fallback is instead observable through the streaming_fallback_* metrics.

Metrics¶

When the metrics feature is active, the fallback collector registers:

• fallback_attempts_total{original_model,fallback_model,backend}
• fallback_success_total{original_model,fallback_model,backend}
• fallback_exhausted_total{original_model}
• models_using_fallback{original_model}
• fallback_duration_seconds{original_model,success}
• fallback_triggers_by_reason{original_model,reason}

Mid-stream recovery also exposes streaming_fallback_* metrics.

Reload behavior¶

Updates to fallback_chains and fallback_policy are consumed by the live fallback service. Changing the fallback service's presence or master enabled state requires restart. Treat changes to the streaming service configuration as described by /admin/config/hot-reload-status and the generated configuration comments.

Related pages¶

• Error Handling
• Circuit Breaker
• Load Balancing