Skip to content

Model Fallback

Model fallback maps a requested model to an ordered list of alternatives. It can retry through that list after configured HTTP errors, timeouts, connection failures, model-not-found results, or unhealthy-backend selection failures.

Configuration

fallback:
  enabled: true
  mid_stream_enabled: true

  fallback_chains:
    gpt-5.4:
      - gpt-5.4-mini
      - claude-sonnet-4-6

  fallback_policy:
    trigger_conditions:
      error_codes: [429, 500, 502, 503, 504]
      timeout: true
      connection_error: true
      model_not_found: true
      backend_unhealthy: true
    max_fallback_attempts: 3
    fallback_timeout_multiplier: 1.5
    preserve_parameters: true

  model_settings:
    gpt-5.4:
      fallback_enabled: true
      notify_on_fallback: true

fallback_chains keys are primary models and values are tried in order. Chains cannot be empty, self-referential, or cyclic. Model names are limited to 128 alphanumeric, ., _, or - characters.

fallback_policy.max_fallback_attempts is 1–10. fallback_timeout_multiplier is 1.0–5.0.

The schema also accepts trigger_conditions.circuit_breaker_open. The proxy now consults circuit-breaker state during selection, but a request rejected by an open circuit is surfaced as a backend_unhealthy trigger, not as the distinct circuit_breaker_open reason, so that specific condition is not emitted by ordinary LLM traffic. Enable backend_unhealthy (and active health checks) to advance the fallback chain when a backend's circuit is open or the backend is unhealthy.

Pre-stream execution

For a non-streaming request or a stream that has not begun, the router:

  1. resolves the primary model and its configured chain;
  2. executes the primary attempt;
  3. classifies an eligible failure;
  4. selects the next model and translates supported parameters when the provider changes;
  5. stops at the first success or at max_fallback_attempts.

Fallback is model-based. Each alternative model is resolved through the normal backend pool, health filtering, access policy, and selection strategy.

Cross-provider translation handles representable fields such as token limits, temperature, sampling, and stop sequences. Provider-only fields without an equivalent may be removed.

Streaming fallback

fallback.mid_stream_enabled controls whether the router buffers an active SSE response so it can recover after an upstream stream failure:

  • true (default): mid-stream fallback and its per-stream buffer are allowed.
  • false: pre-stream fallback remains available, but a failure after SSE output begins is returned as a stream error.

streaming.mid_stream_fallback.enabled is a mode selector, not the buffering switch:

  • true: build a continuation request from accumulated output when enough content exists.
  • false: restart the original request on the fallback model.

Configure streaming.mid_stream_fallback.max_fallback_attempts, min_accumulated_tokens, fallback_delay_ms, and continuation_prompt to tune this path. The stream's total and chunk-interval deadlines still apply.

Response headers

When a model's notify_on_fallback is true (the default) and a fallback succeeds, the response can contain:

X-Fallback-Used: true
X-Original-Model: gpt-5.4
X-Fallback-Model: gpt-5.4-mini
X-Fallback-Reason: error_code_429
X-Fallback-Attempts: 2

The router emits five fallback headers: X-Fallback-Used, X-Original-Model, X-Fallback-Model, X-Fallback-Reason, and X-Fallback-Attempts. Setting a model's notify_on_fallback: false suppresses all of them while still serving the fallback response.

X-Fallback-Attempts reports the total number of model attempts made to satisfy the request, counting the primary attempt. Because the header appears only when a fallback backend produced the response, its value is always at least 2: a value of 2 means the primary model failed once and the first fallback succeeded, and larger values indicate additional failed hops before success. This lets a client distinguish a clean first-hop recovery from a chain that traversed several failing models.

These headers are attached to non-streaming responses. A streaming (SSE) request sends its response headers before the body, so a mid-stream recovery cannot report the attempt count in a header; mid-stream fallback is instead observable through the streaming_fallback_* metrics.

Metrics

When the metrics feature is active, the fallback collector registers:

  • fallback_attempts_total{original_model,fallback_model,backend}
  • fallback_success_total{original_model,fallback_model,backend}
  • fallback_exhausted_total{original_model}
  • models_using_fallback{original_model}
  • fallback_duration_seconds{original_model,success}
  • fallback_triggers_by_reason{original_model,reason}

Mid-stream recovery also exposes streaming_fallback_* metrics.

Reload behavior

Updates to fallback_chains and fallback_policy are consumed by the live fallback service. Changing the fallback service's presence or master enabled state requires restart. Treat changes to the streaming service configuration as described by /admin/config/hot-reload-status and the generated configuration comments.