Circuit Breaker¶

Continuum Router contains a per-backend three-state circuit-breaker implementation (Closed, Open, and HalfOpen) and exposes its state through the Admin API. The ordinary LLM proxy path drives and consults it: the router checks admission after a backend is selected and before dispatch, records each request's success or failure outcome, and excludes an open backend from selection so traffic routes around it. When every compatible backend's circuit is open, the router returns the standard service-unavailable error and participates in configured fallback. Circuit-breaker protection complements active health checks and retry/fallback routing on the inference path rather than replacing them.

State Machine¶

The implemented state machine behaves as follows:

These semantics apply to the normal proxy data plane, provider-native streaming handshakes, Admin operations, and any library caller that uses the circuit-breaker API directly.

Configuration Reference¶

circuit_breaker:
  enabled: true
  failure_threshold: 5
  failure_rate_threshold: 0.5
  minimum_requests: 10
  timeout: 60s
  half_open_max_requests: 3
  half_open_success_threshold: 2
  failure_status_codes: [500, 502, 503, 504]
  timeout_as_failure: true
  sliding_window_size: 60s
  sliding_window_type: count_based  # count_based or time_based

  backends:
    openai-primary:
      failure_threshold: 10
      failure_rate_threshold: 0.4
      timeout: 30s
      half_open_max_requests: 2
      half_open_success_threshold: 2

Per-backend overrides support only the five fields shown under backends. minimum_requests, failure status codes, timeout handling, and sliding-window settings remain global.

The duration fields accept units such as ms, s, m, and h. Use continuum-router config validate config.yaml before startup.

Admin Endpoints¶

When the binary includes the admin feature, the following authenticated endpoints operate on the main circuit-breaker instance:

A circuit-breaker section that is absent or disabled leaves the main instance unavailable. Mutating endpoints then return 503 Service Unavailable; the list endpoint reports "enabled": false.

Example status payload:

{
  "backend": "openai-primary",
  "state": "closed",
  "failure_count": 0,
  "success_count": 0,
  "last_failure_time": null,
  "last_success_time": null,
  "last_transition_time": null,
  "next_retry_time": null,
  "consecutive_successes": 0,
  "half_open_requests": 0,
  "statistics": {
    "total_requests": 0,
    "failed_requests": 0,
    "successful_requests": 0,
    "success_rate": "0.00%",
    "times_opened": 0,
    "times_closed": 0,
    "average_open_duration_ms": 0.0
  }
}

Changes to an existing circuit-breaker configuration are applied to the instantiated state machine through config hot reload. Enabling or disabling the optional component itself should be treated as a restart-time change.

Proxy integration¶

The data plane records an outcome exactly once per dispatch that actually reached a backend:

• A successful response records a success and, on a half-open circuit, advances it toward closing.
• A configured failure status code (failure_status_codes, default 500, 502, 503, 504) records a failure; a timeout records a failure when timeout_as_failure is set; a connection or transport error records a failure unconditionally.
• Status codes not listed in failure_status_codes (including 4xx and 429 by default) record a neutral outcome: an admitted half-open slot is released, and the failure count, the success count, and the sliding window are all left untouched. Client/router errors that never describe an upstream response are ignored; if a path had already admitted, that only releases the slot.

Candidate filtering is read-only and happens during selection, so an open circuit is skipped in favor of a healthy peer before a backend is chosen. Half-open admission is checked only when the request is committed to upstream dispatch. That split matters for preflight and cache-aware surfaces: a selected backend may be inspected for compatibility, cache identity, guardrail input gating, or request translation without consuming a half-open probe slot. Once dispatch begins, an open circuit (still cooling down) or a half-open circuit already at its probe ceiling rejects the request, which the router treats as a retryable, fallback-aware backend-selection outcome so another eligible backend or fallback model can be tried.

Coverage, stated precisely, because a seam that is claimed but absent is worse than one that is known to be missing:

Admission and outcome recording: Chat Completions, Completions, Embeddings, Rerank, sparse embeddings, standard streaming, provider-native streaming (Anthropic, Gemini, Bedrock), Unix-socket streaming, image generation and image edit (including the Gemini image paths), the Responses ingress (/v1/responses, all four conversion strategies plus passthrough and compact), the Chat-Completions-to-Responses bridge, the Anthropic Messages ingress (native, OpenAI-compatible, Responses-backed, Bedrock Runtime, and the web-search emulation paths), Anthropic count_tokens, and native Gemini multimodal embeddings.

Selection-time filtering and dispatch-time admission run together in every selection helper that feeds those paths. This pairing matters: admission alone would turn a servable request into a 503 whenever round-robin landed on an open circuit, while selection-time admission would let local cache hits, policy rejections, or guardrail blocks squat scarce half-open probe slots without ever touching the backend.

Not yet covered: the Responses ingress streaming service (stream_service) and its passthrough streaming variant, and the defensive unrouted stream_with_auto_backend_selection path.

Streaming outcome granularity: the mid-stream fallback relay records terminal completion, stream failure, or client disconnect. Every other streaming path records the handshake status only, so a backend that answers 200 and then drops the body is recorded as a success.

Why a non-listed status is neutral, not a success¶

A response the breaker is not configured to count is evidence that the backend answered, not evidence that it is healthy. Recording it as a success reset failure_count to zero and admitted a success into the sliding window, so a backend that interleaved 429s or 404s with genuine 5xx could never reach the threshold, and a backend answering only 429 looked permanently healthy. The neutral outcome removes both effects while still releasing the half-open probe slot the request occupied.

429 stays out of the default failure_status_codes deliberately. A transient rate limit means the backend is alive and throttling; opening its circuit would take it out of rotation for the whole cooldown and turn throttling into an outage. The router already handles 429 through the retry path, honoring the upstream Retry-After and failing fast on non-transient quota or credit exhaustion. Add 429 to failure_status_codes only when a specific backend's 429s genuinely mean it should be de-routed.

Neutral in half-open, and the cross-endpoint reset¶

HalfOpen is the one state where a neutral outcome still moves the circuit, but it no longer closes on neutral evidence alone. Neutral probes count as half-open progress so a backend whose transport recovered but now answers only 429 does not sit in HalfOpen forever, capped at half_open_max_requests concurrent requests with no timeout to end it. When half_open_success_threshold is reached, the breaker closes only if at least one probe in that threshold window was a genuine success. A neutral-only window returns the circuit to Open with a fresh cooldown. In Closed a neutral outcome still records nothing, which is the asymmetry the fix is built on.

One consequence worth knowing when a single backend serves several endpoints: failure_threshold counts consecutive failures, because a success resets the count to zero. A backend whose chat endpoint is failing while its image endpoint stays healthy therefore may not reach the absolute threshold, since the interleaved image successes keep resetting it. The failure_rate_threshold path still opens the circuit in that case, using the sliding window rather than the consecutive count, which is why both triggers exist and both are enabled by default. Lower minimum_requests if you want the rate path to engage sooner on a mixed-traffic backend.

Circuit breaker vs. health checks¶

The two mechanisms are independent and complementary. The health checker never touches the circuit breaker, and the breaker's Open to HalfOpen recovery is driven by its own cooldown plus live traffic, not by a health probe.

The gap between those two columns is the operational point. Without a breaker, exclusion falls entirely to health checking, so for up to the full health-check window every request still tries the dead backend first. With a fallback chain configured, those requests are all rescued and nothing fails, which is exactly what makes the condition easy to miss: under sustained load the per-request connect attempt builds a backlog and first-token latency stays degraded for the whole window instead of recovering.

The router therefore emits a startup warning when fallback.fallback_chains is configured and no circuit breaker is enabled, quantifying the exposure window from the configured health_checks values:

WARN Fallback chains are configured but no circuit breaker is enabled. A backend that
     dies mid-traffic stays in the per-request rotation for up to 90s
     (health_checks.interval x health_checks.unhealthy_threshold) before health checking
     de-routes it. ...

The breaker remains opt-in: enabling it by default would change request routing for every existing deployment. Silence the warning by adding a circuit_breaker section, not by removing the fallback chain.

Operational Guidance¶

• Configure health_checks to remove unhealthy backends from ordinary model selection.
• Configure retry and fallback for transient failures and model-level alternatives.
• The Admin /admin/circuit/* endpoints report the same per-backend state the data plane uses, so they reflect circuits opened by real traffic.
• Circuit-breaker Prometheus metrics (circuit_breaker_state, circuit_breaker_failures_total, circuit_breaker_successes_total, circuit_breaker_transitions_total) are registered into the /metrics registry and reflect real proxy traffic when the metrics feature and a circuit breaker are both enabled.

Related Documentation¶

• Error Handling
• Load Balancing
• Model Fallback