Circuit Breaker¶

Continuum Router contains a per-backend three-state circuit-breaker implementation (Closed, Open, and HalfOpen) and exposes its state through the Admin API. The ordinary LLM proxy path drives and consults it: the router checks admission after a backend is selected and before dispatch, records each request's success or failure outcome, and excludes an open backend from selection so traffic routes around it.

When every compatible backend's circuit is open, the router returns the standard service-unavailable error and participates in configured fallback. Circuit-breaker protection complements active health checks and retry/fallback routing on the inference path rather than replacing them.

State Machine¶

The implemented state machine behaves as follows:

These semantics apply to the normal proxy data plane, provider-native streaming handshakes, Admin operations, and any library caller that uses the circuit-breaker API directly.

Configuration Reference¶

circuit_breaker:
  enabled: true
  failure_threshold: 5
  failure_rate_threshold: 0.5
  minimum_requests: 10
  timeout: 60s
  half_open_max_requests: 3
  half_open_success_threshold: 2
  failure_status_codes: [500, 502, 503, 504]
  timeout_as_failure: true
  sliding_window_size: 60s
  sliding_window_type: count_based  # count_based or time_based

  # Optional: treat a successful but slow answer as a failure (issue #2964).
  latency_as_failure:
    ttft_ms: 5000              # Streaming: provider dispatch to first byte
    total_ms: 60000            # Buffered: whole-response duration
    minimum_samples: 10        # Judged observations before a breach counts
    model_overrides:
      gpt-5:
        ttft_ms: 15000         # Reasoning models need a longer first-token budget

  backends:
    openai-primary:
      failure_threshold: 10
      failure_rate_threshold: 0.4
      timeout: 30s
      half_open_max_requests: 2
      half_open_success_threshold: 2

Per-backend overrides support only the five fields shown under backends. minimum_requests, failure status codes, timeout handling, sliding-window settings, and latency_as_failure remain global; per-model latency tuning uses the block's own model_overrides map instead.

The duration fields accept units such as ms, s, m, and h. Use continuum-router config validate config.yaml before startup.

Admin Endpoints¶

When the binary includes the admin feature, the following authenticated endpoints operate on the main circuit-breaker instance:

A circuit-breaker section that is absent or disabled leaves the main instance unavailable. Mutating endpoints then return 503 Service Unavailable; the list endpoint reports "enabled": false.

Example status payload:

{
  "backend": "openai-primary",
  "state": "closed",
  "failure_count": 0,
  "success_count": 0,
  "last_failure_time": null,
  "last_success_time": null,
  "last_transition_time": null,
  "next_retry_time": null,
  "consecutive_successes": 0,
  "half_open_requests": 0,
  "statistics": {
    "total_requests": 0,
    "failed_requests": 0,
    "successful_requests": 0,
    "success_rate": "0.00%",
    "times_opened": 0,
    "times_closed": 0,
    "average_open_duration_ms": 0.0
  }
}

Changes to an existing circuit-breaker configuration are applied to the instantiated state machine through config hot reload. Enabling or disabling the optional component itself should be treated as a restart-time change.

Proxy integration¶

The data plane records an outcome exactly once per dispatch that actually reached a backend:

• A successful response records a success and, on a half-open circuit, advances it toward closing.
• A configured failure status code (failure_status_codes, default 500, 502, 503, 504) records a failure; a timeout records a failure when timeout_as_failure is set; a connection or transport error records a failure unconditionally.
• Status codes not listed in failure_status_codes (including 4xx and 429 by default) record a neutral outcome: an admitted half-open slot is released, and the failure count, the success count, and the sliding window are all left untouched. Client/router errors that never describe an upstream response are ignored; if a path had already admitted, that only releases the slot.

Candidate filtering is read-only and happens during selection, so an open circuit is skipped in favor of a healthy peer before a backend is chosen. Half-open admission is checked only when the request is committed to upstream dispatch. That split matters for preflight and cache-aware surfaces: a selected backend may be inspected for compatibility, cache identity, guardrail input gating, or request translation without consuming a half-open probe slot.

Once dispatch begins, an open circuit (still cooling down) or a half-open circuit already at its probe ceiling rejects the request. The router treats that rejection as a retryable, fallback-aware backend-selection outcome so another eligible backend or fallback model can be tried.

Circuit-breaker coverage across proxy routes is organized as follows:

• Full admission and outcome recording:
  • Chat Completions, Completions, Embeddings, Rerank, and sparse embeddings.
  • Standard streaming, provider-native streaming (Anthropic, Gemini, Bedrock), and Unix-socket streaming.
  • Image generation and image edit (including the Gemini image paths) and native Gemini multimodal embeddings.
  • Responses ingress (/v1/responses, all four conversion strategies plus passthrough and compact) and the Chat-Completions-to-Responses bridge.
  • Anthropic Messages ingress (native, OpenAI-compatible, Responses-backed, Bedrock Runtime, and web-search emulation paths) and Anthropic count_tokens on its native Anthropic arm.
• Non-probing admission (outcome recorded for transport failures alone): the engine tokenizer routes (/tokenize, /detokenize, and their /v1 aliases) and the engine tokenize arm of Anthropic count_tokens.
  • Cooldown and slot behavior: An open circuit inside its cooldown refuses these requests. Past the cooldown, admission moves the circuit to half-open as a probe would, and a half-open circuit admits them regardless of probe occupancy without taking a probe slot (Admission::open_non_probing, backed by CircuitBreaker::allow_request_non_probing and admits_selection_non_probing at selection). Tokenizer load therefore never delays the generation probe that can close the circuit.
  • HTTP status neutrality: Any HTTP response from the engine records nothing (no success, no failure, and no half-open progress). A tokenizer answer says nothing about the generation path, and a client can provoke an engine 5xx on these routes (including via count_tokens text).
  • Transport failures: Only a connection failure or a timeout is recorded as a failure (record_non_probing_transport_failure, which counts like any failure and reopens a half-open circuit without releasing a probe slot); a timeout honors timeout_as_failure. A circuit reached only by tokenizer traffic after opening stays half-open until a generation request settles it.
  • count_tokens arm selection: count_tokens admits before knowing which arm the selected backend takes, because re-selection can move it to a backend of another type. When candidates include a non-engine backend, it selects as a probe and the tokenize arm returns the slot before its call (Admission::into_non_probing); when every candidate is an engine tokenizer, it selects and admits non-probing from the start.
• Selection-time filtering and dispatch-time admission: Both run together in every selection helper that feeds the covered paths. Admission alone would turn a servable request into a 503 whenever round-robin landed on an open circuit, while selection-time admission would let local cache hits, policy rejections, or guardrail blocks occupy scarce half-open probe slots without touching the backend.
• Not yet covered: The Responses ingress streaming service (stream_service), its passthrough streaming variant, and the defensive unrouted stream_with_auto_backend_selection path.
• Streaming outcome granularity: The mid-stream fallback relay records terminal completion, stream failure, or client disconnect. Every other streaming path records the handshake status only, so a backend that answers 200 and then drops the body is recorded as a success. The standard OpenAI-wire chat relay additionally applies latency_as_failure at the first-byte seam described below; the other relays still settle at the handshake.

Latency as a failure criterion¶

A backend can be alive, answer every request with a 2xx, and still be unusable because its first token takes longer than the client is willing to wait. failure_threshold and failure_rate_threshold never see that case: the request succeeds, just too late. latency_as_failure makes a slow success count as one terminal failure, so the backend leaves rotation under the same threshold and cooldown as an erroring backend and returns through the same half-open probe.

latency_as_failure:
  ttft_ms: 5000
  total_ms: 60000
  minimum_samples: 10

• ttft_ms bounds the streaming path: the time from provider dispatch to the first upstream byte. A buffered request ignores it, because there is no first byte distinct from the whole answer.
• total_ms bounds the buffered path: the whole-response duration. A streaming request ignores it.
• At least one of the two must be set. Both may be set, and each observation uses only the threshold for its own kind; a streaming TTFT is never compared against total_ms.
• minimum_samples (default 10) is a warm-up measured in judged observations since the circuit last closed, reset, or had its sliding window cleared. In Closed, the first minimum_samples - 1 slow observations are recorded as successes, and only an observation at or past the floor counts as a failure. Half-open probes skip the warm-up: the circuit was opened by real evidence, so a slow probe result reopens it immediately.
• model_overrides maps a served model id to its own optional ttft_ms, total_ms, and minimum_samples. An override replaces only the fields it sets.

The judged duration is partly client-influenced: a large prompt raises prefill time before the first token, and a large max_tokens raises whole-response time. With untrusted clients, raise minimum_samples and the thresholds, or scope the criterion per model with model_overrides, so an expensive request is not mistaken for a degrading backend.

Each dispatch produces exactly one outcome, never a success followed by a failure: the verdict is computed before anything is recorded, so a slow result increments the failure count without first resetting it through record_success. A fast result records an ordinary success. The transition log and the reason label on circuit_breaker_transitions_total report slow, and a slow result is counted on circuit_breaker_failures_total{error_type="slow"}.

Coverage is the standard OpenAI-wire chat path: a buffered reply is judged when the whole response has been read, and a streamed reply defers the handshake outcome to the relay's first-byte stamp. If a deferred stream ends without producing a first byte, the admission is released as neutral rather than counted as either outcome.

The latency criterion does not apply to these surfaces yet:

• The Responses ingress streaming services and their passthrough variants.
• The pre-stream and mid-stream fallback relays.
• Provider-native streaming handshakes (Anthropic, Gemini, Bedrock).
• Unix-socket streaming.
• The image, embedding, rerank, and tokenizer routes.

LeastLatency selection and this criterion are complementary. LeastLatency steers new requests toward the fastest observed backend but never removes one; latency_as_failure removes a backend whose latency has crossed an operator-set bound.

Why a non-listed status is neutral, not a success¶

A response the breaker is not configured to count is evidence that the backend answered, not evidence that it is healthy. Recording it as a success reset failure_count to zero and admitted a success into the sliding window, so a backend that interleaved 429s or 404s with genuine 5xx could never reach the threshold, and a backend answering only 429 looked permanently healthy. The neutral outcome removes both effects while still releasing the half-open probe slot the request occupied.

429 stays out of the default failure_status_codes deliberately. A transient rate limit means the backend is alive and throttling; opening its circuit would take it out of rotation for the whole cooldown and turn throttling into an outage. The router already handles 429 through the retry path, honoring the upstream Retry-After and failing fast on non-transient quota or credit exhaustion. Add 429 to failure_status_codes only when a specific backend's 429s genuinely mean it should be de-routed.

Neutral in half-open, and the cross-endpoint reset¶

HalfOpen is the one state where a neutral outcome still moves the circuit, though it does not close on neutral evidence alone. Neutral probes count as half-open progress so a backend whose transport recovered but now answers only 429 does not sit in HalfOpen indefinitely at half_open_max_requests concurrency. When half_open_success_threshold is reached, the breaker closes only if at least one probe was a genuine success; a neutral-only window reopens the circuit with a fresh cooldown. In Closed, a neutral outcome records nothing.

When a single backend serves several endpoints, failure_threshold counts consecutive failures because a success resets the count to zero. A backend whose chat endpoint is failing while its image endpoint stays healthy may not reach the absolute threshold if interleaved image successes keep resetting it. The failure_rate_threshold path still opens the circuit in that case by using the sliding window rather than the consecutive count; lower minimum_requests to engage the rate path sooner on a mixed-traffic backend.

Circuit breaker vs. health checks¶

The two mechanisms are independent and complementary. The health checker never touches the circuit breaker, and the breaker's Open to HalfOpen recovery is driven by its own cooldown plus live traffic, not by a health probe.

The gap between those two columns is the operational point. Without a breaker, exclusion falls entirely to health checking, so for up to the full health-check window every request still tries the dead backend first. With a fallback chain configured, those requests are all rescued and nothing fails, which is exactly what makes the condition easy to miss: under sustained load the per-request connect attempt builds a backlog and first-token latency stays degraded for the whole window instead of recovering.

The router therefore emits a startup warning when fallback.fallback_chains is configured and no circuit breaker is enabled, quantifying the exposure window from the configured health_checks values:

WARN Fallback chains are configured but no circuit breaker is enabled. A backend that
     dies mid-traffic stays in the per-request rotation for up to 90s
     (health_checks.interval x health_checks.unhealthy_threshold) before health checking
     de-routes it. ...

The breaker remains opt-in: enabling it by default would change request routing for every existing deployment. Silence the warning by adding a circuit_breaker section, not by removing the fallback chain.

Operational Guidance¶

• Configure health_checks to remove unhealthy backends from ordinary model selection.
• Configure retry and fallback for transient failures and model-level alternatives.
• The Admin /admin/circuit/* endpoints report the same per-backend state the data plane uses, so they reflect circuits opened by real traffic.
• Circuit-breaker Prometheus metrics (circuit_breaker_state, circuit_breaker_failures_total, circuit_breaker_successes_total, circuit_breaker_transitions_total) are registered into the /metrics registry and reflect real proxy traffic when the metrics feature and a circuit breaker are both enabled.

Related Documentation¶

• Error Handling
• Load Balancing
• Model Fallback