Model Fallback¶

Model fallback maps a requested model to an ordered list of alternatives. It can retry through that list after configured HTTP errors, timeouts, connection failures, model-not-found results, or unhealthy-backend selection failures.

Configuration¶

fallback:
  enabled: true
  mid_stream_enabled: true

  fallback_chains:
    gpt-5.4:
      - gpt-5.4-mini
      - claude-sonnet-4-6

  # Catch-all for models with no explicit chain above.
  default_chain:
    - claude-haiku-4-5
    - gpt-5.4-mini

  fallback_policy:
    trigger_conditions:
      error_codes: [429, 500, 502, 503, 504]
      timeout: true
      connection_error: true
      model_not_found: true
      backend_unhealthy: true
      context_window_exceeded: false
    max_fallback_attempts: 3
    fallback_timeout_multiplier: 1.5
    preserve_parameters: true
    max_concurrent_dials_per_backend: 50

  model_settings:
    gpt-5.4:
      fallback_enabled: true
      notify_on_fallback: true

fallback_chains keys are primary models and values are tried in order. Chains cannot be empty, self-referential, or cyclic:

• Allowed characters: A model name is 1 to 128 bytes of Unicode letters and digits, -, _, ., /, and :. HuggingFace paths (meta-llama/Llama-3.1-8B-Instruct) and Ollama tags (llama3:8b, hf.co/org/repo:Q4_K_M) are valid as primaries or hops.
• Validation and normalization: Whitespace, control characters, and other punctuation are rejected with an error naming the character and its byte offset. Model names are used verbatim without trimming, case-folding, or decoding.

Default chain¶

default_chain is a catch-all list for a model that has no fallback_chains entry and no model_settings[].custom_fallback_chain. Resolution order is the custom chain, then the model's fallback_chains entry, then default_chain, so an explicit entry always wins and an absent list preserves the previous behavior (a model with no chain gets no failover). The list reloads immediately.

• Flat and shared: The dispatcher's own name is dropped from its resolved list, so a model never hops to itself. The executor never follows a hop's own chain, so the list cannot cycle.
• Validation: Names follow the fallback_chains rule, and a duplicate is refused.
• Key scoping: A default-chain target the request key's allowed_models does not permit is skipped instead of dispatched, logged, and counted under fallback_default_chain_hop_skipped_total{original_model,fallback_model}. Explicit chains keep their existing behavior: they are operator-authored per model and are not re-gated per hop.
• All targets filtered: A request whose every target is dropped behaves as if the model had no chain.
• Carried-over semantics: every unlisted model counts as chained, so a quality_check with an empty models list judges its replies, and the response cache route stays disabled for it, exactly as for an explicitly chained model.

Pair a default chain with model_settings entries for models that must not fail over: without an explicit entry, a default chain enables fallback for every unlisted model.

fallback_policy controls attempt limits, per-hop timeouts, and concurrency bounds:

• max_fallback_attempts (1 to 10): Maximum number of fallback attempts.
• fallback_timeout_multiplier (1.0 to 5.0): Scales each hop's request timeout. The primary attempt (attempt 1) keeps its configured timeout, and hop n runs with base * multiplier^(n-1), clamped to the timeouts.limits ceiling for that timeout class.
  • Base timeout: Uses the attempt's normal timeout: per-model timeouts.request.standard.total on OpenAI-compatible non-streaming routes, the per-attempt streaming window during pre-stream connection, and the per-model profile (total, plus the first-SSE-byte deadline on streaming) on /anthropic/v1/messages and /v1/responses.
  • Unscaled operations: Does not scale timeouts.connection, the wait for a fallback dial permit, a within-hop retry, or a mid-stream hop taken after the client already holds the stream.
  • Streaming budget composition: Distinct from timeouts.streaming_fallback_budget_multiplier, which sizes the shared wall-clock budget across all attempts of a streaming request. The connection phase scales the hop window first and then caps it at the remaining shared budget.
• max_concurrent_dials_per_backend (0 to 10000, 0 = unlimited, default 50): Per-backend ceiling on concurrent fallback hop dials across all fallback paths, held only until the provider handshake returns. See Concurrent request storms for hold scope, saturation behavior, and reload semantics.

The schema also accepts trigger_conditions.circuit_breaker_open. The proxy consults circuit-breaker state during selection, but a request rejected by an open circuit is surfaced as a backend_unhealthy trigger rather than a separate circuit_breaker_open reason. Enable backend_unhealthy (and active health checks) to advance the fallback chain when a backend's circuit is open or the backend is unhealthy.

Context-window fallback¶

trigger_conditions.context_window_exceeded (default false) hops to a chain target before dispatch when the input cannot fit the primary model's declared context window. The ingress admission check already compares the typical input estimate with the primary's effective window. With the trigger on, the router walks the configured chain in order and dispatches the first target whose own declared window holds the same estimate as the primary attempt.

• Opt-in: With the trigger off, an over-long input returns the 400 context_length_exceeded refusal unchanged.
• Input-only judgment: The target window is judged on the input estimate alone, so a request with no max_tokens or max_completion_tokens still hops.
• No fitting target: When every permitted target's window is too small, the client receives the primary's refusal naming its window and the estimated input tokens, and no backend is dialed.
• Fail-open: A model with no known limit, or a declared window of 0, is forwarded and never triggers a hop.
• Attempt budget: At most max_fallback_attempts - 1 targets are considered, because the primary counts as attempt 1.
• Attribution: A dispatched hop sets the usual X-Fallback-* headers with X-Fallback-Reason: context_window_exceeded.
• Guardrail scope: The hop runs before the input guardrail gate, so the target's guardrails.routes policy screens the request, matching the model that actually receives it. An ordinary post-dispatch hop is screened under the primary's route first, so mirror any mandatory route rules across the models you chain.

Pre-stream execution¶

For a non-streaming request or a stream that has not begun, the router:

1. resolves the primary model and its configured chain;
2. executes the primary attempt;
3. classifies an eligible failure;
4. selects the next model, rewrites the payload's model name, and fits chat output-token fields to known target limits;
5. stops at the first success or at max_fallback_attempts.

Fallback is model-based. Each alternative model is resolved through the normal backend pool, health filtering, access policy, and selection strategy.

Chat output-token budgets¶

Every /v1/chat/completions fallback hop fits a requested output-token budget to the target model before opening the provider connection. This applies to buffered execution, pre-stream connection fallback, mid-stream recovery, and response-quality fallback:

• Limit sources and precedence: The router combines explicit metadata.limits from the request's configuration snapshot with the aggregated model catalog. Explicit limits from the request snapshot take precedence over a cached catalog from an older configuration generation, and backend-specific metadata takes precedence over global metadata.
• Snapshot-only lookup: A hop reads only the request snapshot and the existing in-memory catalog (fresh or stale) without triggering discovery or revalidation. Because a fallback hop runs immediately after a backend failure, often inside a shared streaming budget, running discovery would dial the failed backend again.
• Startup catalog warm: Building the router warms the catalog once in the background across all entry points (serve, serve_embedded, and into_router()), regardless of model_aggregation.background_refresh. The warm does not delay startup, shares its fan-out with the background refresher's first pass, and respects model_aggregation.request_timeout, max_retries, and retry_delay. If a hop runs before the first warm finishes (or after all warms fail) and the target has no explicit metadata, the hop payload is left unchanged.

When the request contains max_completion_tokens, max_tokens, or both, each supplied field is capped by the target's known max_output and the target context window minus an estimated input size. Both fields remain present when both were supplied, and existing max_completion_tokens precedence is unchanged. A zero or absent catalog limit means unknown; if neither limit is known, or the request supplied neither field, the payload remains unchanged.

The input size is an estimate rather than an exact tokenizer count. It walks all prompt-bearing message values (name, tool_calls, function_call, and tool_call_id), plus tools, functions, and response_format, adding a fixed margin per message and for the overall chat template:

• Typical estimate (admission check): Counts ASCII text at 3 characters per token and all other characters at 1 token per character, avoiding characters-per-four underestimates on multilingual text. This estimate decides whether the target can accept the prompt at all, avoiding false 503 rejections from overestimation.
• Upper-bound estimate (clamp sizing): Counts 1 token per UTF-8 byte to size the output clamp without triggering an upstream 400. When the upper bound already fills the context window, the clamp uses half of the remaining space from the typical estimate.
• Target skipping and exhaustion: A target whose context window is already consumed by the typical estimate, or that has fewer than 16 output tokens remaining when the request asked for more, is skipped before dispatch. Budgets are only lowered (never raised), integral floats such as 8192.0 are parsed as integers, and the router never invents a 1-token budget.
• Terminal behavior: Exhausting all targets before any successful reply returns a deterministic 503 incompatibility error. Response-quality fallback retains its fail-open rule and serves the latest successful reply when all remaining targets are incompatible.

This protection does not make provider HTTP 400 responses globally retryable. Providers may tokenize differently or enforce unreported limits, and remote image URLs do not reveal multimodal token charges in advance, so HTTP 400 responses follow the configured status-code policy.

Clamp logs record only the target model and numeric budget metadata, never prompt content. The 503 error body omits the target model name because a chain can reference a model the caller's API key cannot request directly. Requests that omit both max_tokens and max_completion_tokens are not clamped, preserving provider-side defaults (such as the Anthropic transform's max_tokens default).

Cross-provider translation¶

A fallback hop payload stays in canonical OpenAI chat-completions form from end to end, and wire-format conversion happens exactly once, at dispatch, keyed on the selected backend's configured backend_type. Messages, tools, tool_choice, stop, and every sampling parameter therefore reach the backend transform exactly as the client sent them, and the transform, which is the only layer that knows which backend actually serves the hop, produces the provider-native request from that canonical input.

Apart from the target-limit adjustment above, the hop swaps the payload's model name for the hop target. Nothing is provider-preconverted or pre-filtered, because the hop can only infer a provider from the model name and that inference is wrong for claude-* models served through Bedrock, Gemini behind its OpenAI-compatible endpoint, gpt-prefixed open models on vLLM, and every operator-chosen alias, which infers no provider at all.

Three Anthropic-native fields ride on a canonical body and are not part of the OpenAI chat-completions schema: speed (fast mode) and the extended-thinking pair thinking and output_config. They are removed at dispatch, keyed on the selected backend's configured backend_type, not at the hop:

• OpenAI-wire backends (generic, openai, azure, gemini, vllm, ollama, llamacpp, mlxcel, lmstudio, sglang): Stripped from /v1/chat/completions bodies at every send site. No OpenAI-wire target reads these field names, and strict cloud endpoints reject unrecognized top-level keys with HTTP 400.
• anthropic and bedrock: Preserved. Their transforms consume thinking and output_config, and speed remains governed by the backend's anthropic_fast_mode opt-in.
• continuumrouter: Preserved. Downstream Continuum Router instances understand these canonical extension fields and apply the same rule against their own backends, preserving reasoning intent when hopping through a downstream router that fronts Anthropic.
• Unresolved backend snapshot: Left untouched only if the selected backend name is absent from the dispatch configuration snapshot (for example, during an in-flight hot reload that removed or renamed the backend). A backend declared without type: defaults to generic and strips these fields like any other OpenAI-wire backend.

reasoning_effort is never stripped, and a value the client sent is never overwritten. It is the provider-neutral spelling every target either honors or ignores harmlessly, so it is what carries a client's reasoning intent across a hop, which is also why the conversion described next writes into that key when the client did not set it. extra_body is never touched either: stripping is top-level only.

The thinking config becomes reasoning_effort¶

Stripping alone is fail-safe but lossy: a request that asked for extended thinking would hop onto a target running with its own default reasoning behavior. The same dispatch-side gate translates before it strips. On exactly the requests that strip, the Anthropic thinking config is mapped to a reasoning level and written as reasoning_effort, and only then are the three fields removed. The mapping is the inverse of the OpenAI-to-Anthropic direction and reuses its tables, so the two stay inverses of each other.

The budget rows are not specific to a hop. The /anthropic/v1/messages ingress resolves budget_tokens through the same function, ReasoningEffort::from_budget_tokens, on both its Chat Completions transform and the Responses API transform it uses for responses_only models, so a budget resolves to the same reasoning level whichever door the request enters through.

output_config.effort takes precedence over thinking, except when thinking.type is disabled (which emits nothing), and an unrecognized effort string falls through to budget_tokens. Values none, minimal, auto, and max are never emitted: some targets reject them, and default reasoning behavior is expressed by omitting the field.

An existing reasoning_effort (or nested reasoning.effort) always takes precedence and is never overwritten, as it represents the client's explicit provider-neutral intent. A malformed thinking config degrades to a plain strip without failing the hop.

Fitting the level to the target¶

The conversion runs at the send site, after the reasoning validation that guards a client-supplied reasoning_effort, so the emitted value is fitted to the selected target's supported vocabulary:

• openai and azure: Consult the OpenAI model capability table:
  • Non-reasoning models (gpt-4o, chat/instant tiers, and unrecognized Azure deployment names): Receive no reasoning_effort, and the hop proceeds as a plain strip.
  • Model-specific vocabularies: gpt-5-pro allows high only; legacy pro tiers allow medium, high, and xhigh; o-series, codex tiers, and unlisted reasoning-capable GPT-5 variants allow low, medium, and high.
  • Closest-level fitting: Allowed levels are used as is. xhigh on a target without xhigh becomes high. Otherwise, the closest allowed level on the low < medium < high < xhigh ladder is chosen (preferring the higher level on a tie, so low becomes high on gpt-5-pro and medium on gpt-5.2-pro).
  • Azure deployment names: The model field is trusted to match the underlying model family, just as in client-supplied reasoning_effort validation. Name Azure deployments after the model family they serve.
• All other OpenAI-wire targets (generic, gemini, and self-hosted engines vllm, ollama, llamacpp, mlxcel, lmstudio, sglang): Receive low, medium, or high, never xhigh.

Local-engine targets receive the derived effort¶

Self-hosted engines are included deliberately. reasoning_effort is the canonical chat-completions field that every target either consumes or ignores, and the inbound /v1/messages handler already generates it toward these engines.

vLLM, SGLang, llama.cpp, Ollama, and LM Studio accept reasoning_effort on their OpenAI-compatible surfaces or ignore unrecognized sampling parameters without failing the request. Cloud Gemini's transform maps it to thinking_level, so Gemini targets benefit directly while self-hosted rescue hops preserve the caller's reasoning intent.

Streaming fallback¶

fallback.mid_stream_enabled controls whether the router buffers an active SSE response so it can recover after an upstream stream failure:

• true (default): mid-stream fallback and its per-stream buffer are allowed.
• false: pre-stream fallback remains available, but a failure after SSE output begins is returned as a stream error.

streaming.mid_stream_fallback.enabled is a mode selector, not the buffering switch:

• true: build a continuation request from accumulated output when enough content exists.
• false: restart the original request on the fallback model.

Configure streaming.mid_stream_fallback.max_fallback_attempts, min_accumulated_tokens, fallback_delay_ms, and continuation_prompt to tune this path. The stream's total and chunk-interval deadlines still apply.

Same-model sibling continuation¶

streaming.mid_stream_fallback.same_model_first (default false) inserts a same-model hop ahead of the chain after a mid-stream failure. Instead of immediately advancing to the next model in fallback.fallback_chains, the relay looks for another backend that serves the same public model:

• Eligible sibling: user-routable, permitted for the API key, servable by the committed OpenAI-wire relay, admitted by its circuit breaker, and not already dialed for this model in the request.
• Hop semantics: the sibling receives the same continuation or restart prompt the relay builds for a chain target, keeps the response model at the client's model, and counts toward max_fallback_attempts.
• Chain fallback: when the sibling set is empty, or the sibling attempt also fails, the chain advances exactly as it does with the flag off.

The attempt tree records the superseded attempt as fallback_source on the old backend and the sibling as a normal successor under the same request_id. streaming_fallback_total{reason="same_model_sibling"} counts the hop. Mid-stream recoveries cannot carry X-Fallback-* headers because the SSE response is already committed, so the headers do not distinguish a sibling hop from a model change; the flag applies only to the OpenAI-wire mid-stream relay.

Cross-provider streaming hops¶

A streaming fallback hop re-enters the same typed per-backend dispatch direct routing uses, keyed on the selected backend's configured backend_type. A hop onto a native Anthropic, Bedrock, or Gemini backend is served through that provider's native streaming pipeline (for Anthropic, the Messages API at /v1/messages with x-api-key and anthropic-version), and the client keeps receiving OpenAI-format SSE. Configuring a chain for a model whose primary backend is native does not affect healthy direct streaming to it.

The primary attempt uses the same dispatch path regardless of wire protocol:

• Pre-handshake failures on native primaries: A native primary (anthropic, gemini, or a bedrock Runtime/Converse endpoint) that fails before the provider handshake (refused connection, transport error, handshake timeout, or circuit rejection) enters the pre-stream chain walk under both mid_stream_enabled modes and attaches the standard X-Fallback-* headers on recovery.
• Non-terminal native hops: Chain entries that resolve to native backends advance to the next entry on failure just like OpenAI-wire entries.
• Provider status codes: Answered Gemini or Bedrock HTTP statuses follow the OpenAI-wire policy: non-429 4xx responses are terminal, while 429 and 5xx advance only when listed in fallback.fallback_policy.trigger_conditions.error_codes. Refused connections follow trigger_conditions.connection_error.
• Anthropic and Unix-socket specifics: Anthropic native responses retain their existing status handling. A native primary reached over a Unix socket is served as a single unit, so a failure there is returned directly to the client.

The recovery point decides what a hop can reach:

• Before SSE commit (initial connection and pre-stream hops): Any backend type can fail or serve as a hop target in both mid_stream_enabled modes. Because the response is committed only at the first successful provider handshake, pre-stream exhaustion returns a standard HTTP error rather than a 200 stream carrying an error event, and X-Fallback-* headers can still be attached.
• After SSE output begins (mid-stream recovery): Mid-stream recovery uses only the OpenAI-compatible wire after response headers are already sent. Chain entries that resolve to native-protocol or Unix-socket backends are skipped deterministically in favor of the next entry, and streams served through a native pipeline surface mid-stream failures as normal stream errors.

The recovery phase also determines which limits and timeouts apply:

• Attempt limits: Pre-stream hops use fallback.fallback_policy.trigger_conditions and fallback.fallback_policy.max_fallback_attempts, while streaming.mid_stream_fallback.max_fallback_attempts bounds only hops performed by the committed relay.
• Connection budget and keep-alive: Because the response commits at the first successful handshake, no SSE keep-alive comments are sent while connecting. The connection phase shares a single cross-attempt budget with the relay (timeouts.request.streaming.total multiplied by timeouts.streaming_fallback_budget_multiplier), caps each handshake attempt at the remaining budget, and returns the preserved upstream failure once the budget expires. Once a handshake succeeds, the relay measures its read window from that moment.

Ingress coverage¶

A fallback chain is keyed on the dispatched model name and runs on every ingress that resolves a backend from that name:

• OpenAI-shaped ingresses (/v1/chat/completions, /v1/completions, /v1/embeddings, rerank, sparse embeddings, and /v1/images/generations): Execute the chain through the shared dispatch funnel.
• Provider-shaped ingresses (/anthropic/v1/messages and /v1/responses): Dispatch natively and run the same chain through per-attempt selection and dispatch, sharing trigger_conditions, max_fallback_attempts, and max_concurrent_dials_per_backend with chat.
• Excluded endpoints: /v1/responses/compact and count_tokens do not run fallback chains.

On /anthropic/v1/messages and /v1/responses, each attempt preserves the ingress wire format:

• Wire fidelity: The primary attempt sends the client's native Anthropic Messages or Responses payload, and each hop swaps only the model name before converting at dispatch according to the selected backend's backend_type (covering both passthrough and convert strategies on /v1/responses).
• Outcome classification: A 2xx response commits the attempt. Refused connections, timeouts, unknown models, and unhealthy backends trigger hops under connection_error, timeout, model_not_found, and backend_unhealthy. Provider HTTP status codes trigger a hop only when listed in trigger_conditions.error_codes.
• Status-mapped failures: Router-side total timeouts (504) and convert-strategy connection failures (502) are matched against error_codes (both are included by default).
• Error dialect: When the chain is exhausted or fails on a non-trigger status (such as 400), the client receives the last attempt's error in the ingress's native format.

On the streaming arms of both provider-shaped ingresses, fallback hops occur only before the first byte. A failure before the provider handshake advances to the next chain entry, commits the response at the first successful handshake, and attaches X-Fallback-* headers. Failures after the handshake are returned to the client without mid-stream recovery.

Answer-quality fallback¶

Every trigger above is a failure the transport or the provider reports. A backend that answers 200 with a refusal, an empty or cut-off reply, or an answer to a different question is served as is.

fallback.quality_check closes that gap for non-streaming /v1/chat/completions replies: a System One decision model (TypeSafe's hosted Jev, or a self-hosted laya-serve, Kev, or Nimble server, see Decision Models) judges the completed reply, and when it is probably unusable the router advances the same chain it would have advanced on a failure.

fallback:
  fallback_chains:
    qwen3.8-8b: [qwen3.8-27b]
  quality_check:
    target:
      endpoint: "http://laya.internal:8000/v1/systemone"
    threshold: 0.8        # hop when 1 - answered, refusal, or truncated reaches it
    max_extra_hops: 1     # quality hops per request
    models: [qwen3.8-8b]  # chain primaries to judge; empty means every chained model
    mode: shadow          # the default: judge and record, never hop; enforce hops

The block is off when absent, and target is the shared System One target block described in the configuration assistant (endpoint plus an optional api_key_env, or a backend: reference to a self-hosted OpenAI-wire backend).

Start in shadow¶

mode defaults to shadow, like every decision-model consumer: the 0.8 threshold is an uncalibrated starting value, and false positives change the reply a client receives. In shadow mode, the router judges every eligible reply, logs the three probabilities at debug level, and increments fallback_quality_checks_total{outcome="shadow"} whenever enforce would have hopped, while still serving the first reply.

Before switching to mode: enforce, compare shadow against passed to measure the trigger rate, inspect a sample of would-be hops, and tune threshold until triggers match replies you want to replace. See the calibration procedure in Decision Models.

What is judged¶

One decision call asks three noul questions over a state of exactly two texts. When usage.quality_judgment shares the call (same target and caps, see Decision Models), the call over the original reply also carries that judgment's language_match as a best-effort question: a missing or malformed answer to it is left out and never changes the verdict below.

When more than one reaches the threshold, the most probable one is reported. A reply whose finish_reason is length stopped at the client's own max_tokens; a hop with the same limit would be cut the same way, so its truncated answer is logged but never triggers.

The state is {user_message, ai_response}, containing the last user message text and the assistant reply text (or refusal field). Each text is capped at max_request_bytes and max_reply_bytes by keeping both head and tail so the end of a long reply remains visible for truncation checks.

The serialized state is then fitted under target.max_state_bytes so JSON- or code-heavy replies are judged on a shorter excerpt rather than rejected. System/developer prompts, earlier turns, tool calls/results, images, and reasoning text are never sent to the decision model, and tool-calling replies are not judged.

The caps are sized per checkpoint:

How a quality hop runs¶

• Guardrails first. The check runs only on a reply every guardrail stage let through: an input Block returns before dispatch, and an output Block returns before the check. A blocked request or reply is never sent to the decision model. A hop's reply goes through the same output gate before it can be served: a Block leaves the previous reply in place, and a monitor-mode Flag is recorded and served.
• Same chain, same bounds. The hop is the next model of the chain the reply was served from, dispatched through the same hop path the chain executor uses: it counts against fallback_policy.max_fallback_attempts, runs with its own fallback_timeout_multiplier window, takes a fallback dial permit (max_concurrent_dials_per_backend), and honors the key's allowed_backends and model_settings.<model>.fallback_enabled.
• The key's model allow-list is re-applied. A transport-triggered hop skips the key's allowed_models check because the operator's chain chose the target, but a quality hop is client-steerable: a client can deliberately elicit a refusal or a non-answer. So before judging, the router checks the next chain model against the key's allowed_models; when it is not allowed the check is skipped (fallback_quality_checks_total{outcome="skipped"}) and the original reply is served.
• A refusal trigger resubmits a declined request. When the first model refused, a quality hop sends the same request to the next model in the chain. Whether a request one model declined should be retried on another is an operator decision with safety implications: leave a model out of models, or put a more conservative model next in its chain, where that is not wanted. Guardrails still apply to every hop, input and output alike.
• One chain budget. The deadline is the primary model's timeouts.request.standard.total multiplied by timeouts.streaming_fallback_budget_multiplier (default 2.0), measured from the start of dispatch. Transport attempts use their own per-attempt windows, and the quality check and its hops use only the remaining budget. If no budget remains, the reply is served unjudged (outcome="budget_spent").
• Never more than max_extra_hops. A reply is judged only when a further hop is possible, so no decision is paid for that nothing could act on. The reply of the last permitted hop is served without another judgment.
• Fail-open. A decision call that fails or times out, a hop that errors or overruns the budget, and a hop reply the output guardrail blocks all serve the latest reply that already passed the gate: the original, or an earlier hop's when max_extra_hops is 2 or more.

What the client and the operator see¶

A dispatched quality hop sets x-continuum-quality-fallback: 1 on the response. When a hop's reply is served, the X-Fallback-* headers below report it with X-Fallback-Reason: quality_check (subject to notify_on_fallback); when the original is served after a failed hop, only x-continuum-quality-fallback is present.

The decision call's own usage is router-internal: it is logged and never counted toward caller budgets, and target.count_toward_budgets: true is ignored (config validate emits a warning).

Both the judged reply and the quality hop represent real upstream spend and are billed separately: a triggered request produces two request-statistics rows and, on control-plane builds, two usage records with distinct idempotency keys. Request counts for a judged model can therefore reach up to twice the client request count, and the hop row records the hop's own latency.

Not covered: streaming responses, the web-search tool loop (which never takes a chain hop), a reply served from the response cache, and the /anthropic/v1/messages and /v1/responses ingresses. When a hop's reply is served, it is also the reply the response cache stores.

Empty-reply retry¶

A backend can also answer normally with a completion that carries no text at all. fallback.empty_reply_retry covers that case without a decision model: a non-streaming /v1/chat/completions reply whose first choice has no message, or whose message.content (else message.refusal) is missing or whitespace only, advances the same chain.

fallback:
  fallback_chains:
    qwen3.8-8b: [qwen3.8-27b]
  empty_reply_retry:
    enabled: false       # the default: selection and serving are unchanged
    max_extra_hops: 1    # hops this trigger may take per request, 1-5

A request the quality check covers skips this trigger: that check already treats an empty reply as a refusal, so running both would judge and hop on the same reply. Two reply shapes are never empty here:

• A non-empty tool_calls array (or a legacy function_call): a tool call is a complete reply with no text.
• A reply whose finish_reason is length: the client's own max_tokens stopped it, and a hop with the same limit would be cut the same way.

How an empty-reply hop runs¶

• The output gate runs first. The trigger inspects only the body the guardrail output gate passed, never one the gate replaced. A hop's reply passes the same gate before it can be served.
• Same chain, same bounds. The hop uses the quality hop path: it continues the chain's attempt numbering, counts against fallback_policy.max_fallback_attempts, runs with its own fallback_timeout_multiplier window, takes a fallback dial permit (max_concurrent_dials_per_backend), and honors the key's allowed_backends. The key's allowed_models is re-applied to every target.
• One chain budget. The deadline is the primary model's timeouts.request.standard.total multiplied by timeouts.streaming_fallback_budget_multiplier, measured from the start of dispatch.
• Repeat while empty. A still-empty hop reply is examined again and advances the chain while max_extra_hops and the attempt bound allow another hop. The hop target is preflighted against the carried request, so a target whose context leaves no output capacity is skipped without a dial.
• Fail-open. A hop that errors, overruns the budget, or whose reply the output guardrail blocks leaves the latest gate-passed reply in place, empty or not.
• Scope. Only a buffered request with one choice, no log probabilities, and a plain or json_object response format is in scope, served by an OpenAI-wire backend over HTTP(S). A unix: backend, a native azure, anthropic, gemini, or bedrock backend, and a backend with replay: disabled keep their existing behavior.

A served hop reports X-Fallback-Reason: empty_reply_retry through the X-Fallback-* headers (subject to notify_on_fallback). The trigger counts fallback_empty_reply_retry_total{result}, result one of hopped (a hop's reply was served), triggered (a hop was dispatched but failed or was blocked), skipped_no_hop (no permitted further target), budget_spent, and not_enforced (the serving backend is outside the scope above, such as a native provider, a unix: socket, or replay: disabled), and logs one debug line per dispatch with the original model, the serving backend, and the hop model.

Content-policy fallback¶

An input guardrail block is terminal by default: the input gate runs before any dispatch, and its verdict ends the request. fallback.fallback_policy.content_policy_fallback optionally lets that verdict advance fallback_chains instead, so a strict policy on the primary does not reject a request a laxer hop would accept.

fallback:
  fallback_chains:
    qwen3.8-8b: [qwen3.8-27b]
  fallback_policy:
    content_policy_fallback: true   # default false: blocks stay terminal

The trigger is off by default, and an absent field behaves exactly as the contract above. It applies to non-streaming POST /v1/chat/completions. Streaming requests, the classifier egress gate (which runs before a routed model is known), the responses bridge, web-search loop requests, and every other ingress keep the terminal block. So does a request that opts out of cross-model hops, with disable_fallbacks: true (issue #2347) or a fallback: within_variant experiment: its block stays terminal.

How a content-policy hop runs¶

• The hop model's own input gate runs first. The client's document is re-gated under the hop model's guardrails.routes entry (or the global policy) before dispatch, so an input one policy refused never reaches a backend under another model's name. A Transform verdict masks that copy in place.
• Same chain, same bounds. The hop is the next model of the dispatched model's chain, sent through the same hop path a quality hop uses. It counts against fallback_policy.max_fallback_attempts, runs with its own fallback_timeout_multiplier window, takes a fallback dial permit (max_concurrent_dials_per_backend), and honors the key's allowed_backends and model_settings.<model>.fallback_enabled. A client models list (issue #2403) replaces the chain for that request, and a hop target whose backend would run the web-search tool loop is skipped.
• The key's model allow-list is re-applied. Unlike a transport hop, a block-triggered hop is client-steerable, so the next chain model is checked against allowed_models before its gate runs. A disallowed target leaves the primary's block in place.
• One trigger per request. If the hop model's policy also blocks, the hop's own block is served and the chain stops.
• One chain budget. The deadline is the primary model's timeouts.request.standard.total multiplied by timeouts.streaming_fallback_budget_multiplier, measured from the moment the trigger armed. A hop that starts with no budget left is not dispatched, and the block is served.
• One input decision. The primary pass and every hop's re-run pass fold into one request-level decision, so guardrail_verdicts_total and the audit log count the request once. The hop passes record the effective decision.
• Output gate. The hop's reply passes the hop model's output gate before it can be served. A Block from that gate is served instead of the generated reply.
• A served reply is judged by the quality check. The hop-served reply goes through fallback.quality_check like any other non-streaming reply, which may advance the chain once more, and through usage.quality_judgment when configured.
• A failed hop advances, unless the failure is terminal. A transport failure moves to the next chain member under the same attempt budget. A hop that may have reached a replay: disabled backend, or an internally streamed dispatch that already generated output, is final and does not dispatch another model (issues #1993, #2043). When no member can answer, the primary's block is served.

What the client and the operator see¶

A hop that serves a reply attaches the X-Fallback-* headers with X-Fallback-Reason: content_policy (subject to notify_on_fallback). The trigger counts fallback_content_policy_hops_total{result} and records the hop in the ordinary fallback families under the trigger reason content_policy.

Usage attribution¶

Usage is recorded under the model that generated the served reply rather than the model the client originally requested:

• Attributed counters and records: When the primary fails and a fallback target serves the reply, tokens are attributed to the target across /admin/stats, per-model and per-key token counters, model-experiment attribution, and (control-plane builds) the usage record's model field (matching backend and backend_id).
• Covered ingresses and phases: Applies regardless of notify_on_fallback across buffered /v1/chat/completions, /v1/completions, /v1/embeddings, rerank, and sparse-embeddings routes, as well as streaming chat hops taken at selection, pre-stream, or mid-stream. Image endpoints record no token usage.
• Canonical model ID: The recorded model name is always the router's configured ID for the chain target rather than the provider's response model string, preventing split rows across spelling variants.

Per-attempt records¶

Every backend dispatch that reaches a backend is recorded once as an attempt of the client request (covering buffered retries, fallback-chain hops, pre-stream streaming hops, quality-check hops, and each turn of a web_search tool loop):

• Record linkage (control-plane): Each attempt produces a separate usage record linked by a shared request_id, a 1-based attempt_seq, and parent_attempt (the attempt it replaced or continued). The final attempt is recorded by the standard request-level recorder.
• Failed attempts: Recorded immediately before the next retry or hop with outcome: upstream_error (or timeout_first_byte when no byte arrived before timeout), the attempt's http_status (omitted on connect error or timeout), and any tokens reported in the upstream error body.
• Missing backend usage: When a backend reports no usage, the record stores zero output tokens, token_source: none, and a router input-token estimate (estimated: true, capped at 32,768 tokens so base64 images do not inflate the count). Estimates are written only to the usage record; /admin/stats and Prometheus counters remain engine-reported.
• Unreached dispatches: Dispatches rejected locally by the router (invalid backend URL, transform error) do not count as attempts. Requests that never reach any backend (per-key denial, all backends unhealthy) record no usage and appear in /admin/stats under not_dispatched.

For example, a buffered request whose first attempt gets a 503 from backend A, whose retry gets a 429 from backend B, and which backend C then serves, produces three records:

/admin/stats counts that as one request (total_requests: 1, successful_requests: 1) and three attempts (attempts_total: 3, failed_attempts: 2, successful_attempts: 1). A quality-check hop is likewise one more attempt of the same request, not a second request. The streaming relay stamps its own stream-end record's attempt identity; a pre-stream hop leaves the dialed attempt as the request ledger's current one for it.

Mid-stream hops¶

A mid-stream fallback hop splits one client stream across attempts while still recording each backend-reaching attempt once:

• Partial output attribution (fallback_source): When attempt A breaks after streaming output and the relay continues on attempt B, A's record captures the output A generated before the break (counted by the router if the backend sent no usage) with outcome: fallback_source. B's record captures only B's own output.
• Failed attempts: An attempt that generated zero output before replacement records outcome: upstream_error with zero output. A terminal failed attempt with no successor (chain/budget exhausted or client disconnect) records timeout_first_byte, timeout_mid_stream, or upstream_error.
• Sequence numbering: Each dialed hop increments attempt_seq and sets parent_attempt to the attempt it continued under the same request_id (continuing from any pre-stream hop sequence). Hops refused before dispatch (saturated dial bound or open circuit) do not create an attempt.
• Metrics, quality judgments, and export: streaming_fallback_total labels each hop with the failed backend. Any usage.quality_judgment attaches only to the attempt that served the final reply, and GET /admin/usage/export exports one row per attempt.

A plain chain hop stamps no requested_model, requested_provider, or substitution_rule_id, because those fields belong to Hub policies. When a Hub substitution, arbitrage, or budget fallback rewrote the request before a chain hop, requested_model retains the Hub policy's value and model records the final served model.

Model-budget admission checks the pool of the client-requested offering, settlement charges the pool that actually served the request, and the next Hub budget snapshot reconciles the local ledger. /anthropic/v1/messages and /v1/responses chains are not yet covered by per-attempt usage records.

Response headers¶

When a model's notify_on_fallback is true (the default) and a fallback succeeds, the response can contain:

X-Fallback-Used: true
X-Original-Model: gpt-5.4
X-Fallback-Model: gpt-5.4-mini
X-Fallback-Reason: error_code_429
X-Fallback-Attempts: 2

The router emits five fallback headers: X-Fallback-Used, X-Original-Model, X-Fallback-Model, X-Fallback-Reason, and X-Fallback-Attempts. Setting a model's notify_on_fallback: false suppresses all of them while still serving the fallback response.

X-Original-Model names the model the client asked for. When the request named a metadata alias that the router dispatched under its canonical id, the header carries the alias the client sent, not the canonical id the chain was keyed on; X-Fallback-Model still names the model that actually served.

X-Fallback-Attempts reports the total number of model attempts made to satisfy the request, counting the primary attempt. Because the header appears only when a fallback backend produced the response, its value is always at least 2: a value of 2 means the primary model failed once and the first fallback succeeded, and larger values indicate additional failed hops before success. This lets a client distinguish a clean first-hop recovery from a chain that traversed several failing models.

These headers are attached to non-streaming responses and to streaming responses whose fallback occurred before any SSE output (selection-time fallback or pre-stream chain advance in both mid_stream_enabled modes), across all chain-enabled ingresses including /anthropic/v1/messages and /v1/responses.

Because SSE responses send HTTP headers before the body starts, mid-stream recoveries cannot report attempt counts in response headers; monitor streaming_fallback_* metrics instead.

Metrics¶

When the metrics feature is active, the fallback collector registers:

• fallback_attempts_total{original_model,fallback_model,backend}
• fallback_success_total{original_model,fallback_model,backend}
• fallback_exhausted_total{original_model}
• models_using_fallback{original_model}
• fallback_duration_seconds{original_model,success}
• fallback_triggers_by_reason{original_model,reason}
• fallback_default_chain_hop_skipped_total{original_model,fallback_model}: fallback.default_chain targets skipped from a request's resolved chain because the key's allowed_models does not permit them, counted once per request even when the chain is materialized on several phases

Mid-stream recovery also exposes streaming_fallback_* metrics.

The answer-quality check adds three families with closed label sets, and records a quality hop in the families above under the trigger reason quality_check:

• fallback_quality_triggered_total{reason}: quality hops dispatched, reason one of not_answered, refusal, truncated
• fallback_quality_checks_total{outcome}: replies examined, outcome one of passed, triggered, shadow, error, skipped, budget_spent
• fallback_quality_hops_total{result}: dispatched hops by what they produced, result one of served, failed, blocked

The decision calls themselves are counted by the shared client under systemone_requests_total{consumer="fallback_quality"}.

The empty-reply trigger adds one family with a closed label set:

• fallback_empty_reply_retry_total{result}: examined empty replies, result one of hopped, triggered, skipped_no_hop, budget_spent, not_enforced

The content-policy trigger adds one family with a closed label set, and records its hop in the families above under the trigger reason content_policy:

• fallback_content_policy_hops_total{result}: trigger outcomes, result one of triggered (the trigger fired), hopped (a hop passed its own gate and was dispatched), no_chain (no permitted chain target), blocked_again (the hop model's gate blocked, and its block is served), budget_spent (the chain budget was gone)

Side-effecting backends¶

A backend with replay: disabled is never invoked twice for the same request. If an attempt on such a backend fails after the request may have reached it, that failure is terminal across all drivers:

• Non-streaming funnel: Reports TriggerReason::ReplayDisabled, which never triggers a retry or fallback hop.
• Streaming paths: Both the pre-stream connection phase and the mid-stream relay stop chain traversal at that backend. If initial streaming selection lands on a replay: disabled backend, it dispatches once with no chain walk.
• Provider-shaped ingresses and quality checks: /v1/responses and /anthropic/v1/messages chains stop on any failure after committing to the backend, and quality hops never replace a reply served by it.

A chain may still hop onto a replay: disabled backend from an earlier replayable backend. See Replay Policy.

An internally streamed buffered dispatch (proxy.internal_streaming) that fails after its backend generated output is final the same way: the non-streaming funnel reports the attempt as TriggerReason::PartialOutput (partial_output), which should_trigger never accepts, so neither the retry loop, a chain hop, nor the disaggregated unified fallback re-sends it. Failures before any output retries and hops normally; see Internal streaming for buffered requests.

Reload behavior¶

Updates to fallback_chains, fallback_policy, quality_check, and empty_reply_retry are consumed by the live fallback service; quality_check and empty_reply_retry are read from the configuration each request runs under, and a changed quality_check.target rebuilds the cached decision client on the next request. Changing the fallback service's presence or master enabled state requires restart. Treat changes to the streaming service configuration as described by /admin/config/hot-reload-status and the generated configuration comments.

Related pages¶

• Error Handling
• Circuit Breaker
• Load Balancing