Load Balancing Guide¶

When routing a request, Continuum Router first filters the backend pool down to eligible candidates (backends that are enabled, healthy, permitted for the caller's API key, not excluded by an open circuit breaker or retry state, and capable of serving the resolved model). It then evaluates registered scorers (such as the KV cache overlap scorer) when configured, and otherwise applies the top-level selection_strategy.

Canonical configuration¶

selection_strategy: RoundRobin

backends:
  - name: gpu-a
    url: http://gpu-a:8000
    weight: 3
    models: [llama-3.3-70b]
  - name: gpu-b
    url: http://gpu-b:8000
    weight: 1
    models: [llama-3.3-70b]

Configure backend selection using top-level selection_strategy, backends[].weight, backends[].models, smart_routing, prefix_routing, and routing.engine_load. (Note: legacy routing.strategy keys under the routing: block have no runtime effect; always set selection_strategy at the root of config.yaml.)

CLI and environment overrides¶

You can override the startup strategy from the command line or via an environment variable:

# CLI flag
continuum-router --config config.yaml --selection-strategy LeastLatency

# Environment variable (case-insensitive; accepts PascalCase, snake_case, or kebab-case)
export CONTINUUM_SELECTION_STRATEGY=LeastLatency
continuum-router --config config.yaml

Order of precedence (highest to lowest):

1. --selection-strategy CLI flag
2. CONTINUUM_SELECTION_STRATEGY environment variable
3. selection_strategy in config.yaml
4. Built-in default (RoundRobin)

Passing an unrecognized strategy name in CONTINUUM_SELECTION_STRATEGY fails startup with an explicit validation error rather than silently falling back.

Selection strategies¶

RoundRobin (default)¶

selection_strategy: RoundRobin

Cycles through eligible backends in order using an atomic counter. Best suited for homogeneous, stateless backends; ignores backend weights and observed latency.

WeightedRoundRobin¶

selection_strategy: WeightedRoundRobin

backends:
  - name: large
    url: http://large:8000
    weight: 3
  - name: small
    url: http://small:8000
    weight: 1

Distributes requests across eligible backends in proportion to their configured weight. The selector samples a weighted target per request rather than enforcing a strict deterministic 3, 3, 3, 1 sequence. If all eligible candidates have weight: 0, selection falls back to uniform RoundRobin.

LeastLatency¶

selection_strategy: LeastLatency

Selects the eligible backend with the lowest observed average response time. Candidates without prior latency observations are selected via the round-robin counter until measurements are recorded. Because this strategy routes strictly by observed latency, it does not account for prompt length or complexity and can concentrate traffic on a single fast backend.

This strategy steers but never removes: a consistently slow backend keeps its share of traffic as long as it is eligible. To de-route one, enable circuit_breaker.latency_as_failure, which counts a success slower than ttft_ms (streaming) or total_ms (buffered) as a failure and lets the circuit open and later probe it back in.

Random¶

selection_strategy: Random

Selects an eligible backend uniformly at random for each request. Long-run distribution is uniform, though short bursts can be uneven.

ConsistentHash¶

selection_strategy: ConsistentHash

Hashes the resolved model name onto a consistent-hash ring so requests for the same model and candidate set consistently land on the same backend. Note that this provides model affinity, not session or user affinity; if traffic is dominated by a single model name, load will concentrate on one backend.

PrefixAwareHash¶

selection_strategy: PrefixAwareHash

prefix_routing:
  enabled: true
  max_prefix_length: 1024
  load_factor_epsilon: 0.25

Routes requests sharing a common prompt prefix to the same backend using Consistent Hashing with Bounded Loads (CHWBL):

• Prefix key derivation: Computes a SHA-256 hash from model + system_prompt (or model + first_user_message when no system prompt is present) across chat, /v1/responses, and /anthropic/v1/messages requests.

• Bounded-load overflow: When the preferred backend's in-flight request count reaches the CHWBL cap, traffic overflows to the next backend on the hash ring:

  ceil((total_in_flight + 1) * (1 + epsilon) / backend_count)
  

  Lower load_factor_epsilon values prioritize even load distribution; higher values prioritize prefix cache affinity. - Requirement (prefix_routing.enabled: true): When no prefix key can be extracted (or when prefix_routing.enabled is false), selection falls back to model-name hashing without a bounded-load cap, sending all traffic for a given model to a single backend. Always enable prefix_routing.enabled: true when using PrefixAwareHash (config validate emits a warning if it is disabled). - Ring sizing and Anthropic cache control: - prefix_routing.virtual_nodes (default 150 replicas per backend) controls consistent-hash ring granularity and rebuilds the ring on hot reload. - prefix_routing.anthropic_cache_control_injection (default false) acts as a fleet-wide master switch that forces automatic Anthropic cache_control breakpoint injection on eligible native Anthropic requests even when an individual backend sets anthropic_auto_cache_control: false (while preserving any client-supplied cache_control blocks).

EngineLoad¶

selection_strategy: EngineLoad

engine_stats:
  enabled: true

routing:
  engine_load:
    balance_abs_threshold: 64
    balance_rel_threshold: 1.5
    base_strategy: RoundRobin

Routes requests to the candidate reporting the lowest live engine load, using the same engine statistics (waiting_requests and kv_cache_usage) as Engine-load-aware selection.

Whereas the KV scorer's engine-load term ranks only prefix-holding backends when KV cache indexing is active, selection_strategy: EngineLoad ranks the entire eligible candidate set and requires neither prefix_routing nor kv_cache_index.

Each selection pass produces one of three outcomes in routing_engine_load_decisions_total{backend,reason}:

• Ranked (reason="strategy_engine_load"): Every eligible candidate has a fresh engine-stats snapshot and the waiting_requests spread across candidates exceeds both balance_abs_threshold and balance_rel_threshold. The least-loaded candidate is selected.
• Stale fallback (reason="strategy_stale_fallback"): At least one eligible candidate lacks a fresh snapshot, so routing.engine_load.base_strategy decides. Requiring fresh data from all candidates prevents partial statistics on a mixed fleet (e.g., self-hosted vLLM plus a hosted API) from funneling all traffic onto whichever backends publish metrics.
• Hysteresis hold (reason="strategy_hysteresis_hold"): All candidates have fresh snapshots, but the queue spread sits within the hysteresis dead band (balance_abs_threshold / balance_rel_threshold), so base_strategy decides. Because engine snapshots refresh only once per engine_stats.interval, falling back to base_strategy (default RoundRobin, or WeightedRoundRobin on heterogeneous hardware) prevents herd oscillations when queues are nearly balanced.

Configuration notes for selection_strategy: EngineLoad:

• Selecting selection_strategy: EngineLoad activates the strategy directly; it does not require routing.engine_load.enabled: true (which gates only the KV overlap scorer term).
• Requires engine_stats.enabled: true (config validate warns if disabled).
• routing.engine_load.base_strategy cannot be set to EngineLoad itself.
• Shares the same staleness window (engine_stats.interval * routing.engine_load.max_staleness_intervals) and optional saturation admission gate (routing.engine_load.admission) described below.

LeastBusy¶

selection_strategy: LeastBusy

Selects the eligible backend with the fewest requests the router itself has in flight against it, measured by the pool's per-backend in-flight tracker. The signal is fed by every dispatch, so the strategy works over any backend type and requires no engine statistics; a hosted API backend is ranked like a self-hosted one.

Ties, including the all-idle case, rotate through the pool counter, so equally loaded candidates share traffic evenly. Candidates the router has never routed to count as idle.

Because a long streaming request keeps the backend busy until its stream ends, this strategy spreads bursts of slow streams across the pool; LeastLatency would concentrate them on whichever backends answered fastest historically.

LeastCost¶

selection_strategy: LeastCost

backends:
  - name: provider-endpoint
    url: https://api.provider.example/v1
    models: [llama-3.3-70b]
    model_configs:
      - id: llama-3.3-70b
        metadata:
          pricing:
            input_tokens: 2.50
            output_tokens: 10.00
  - name: self-hosted
    url: http://gpu-a:8000
    models: [llama-3.3-70b]
    model_configs:
      - id: llama-3.3-70b
        metadata:
          pricing:
            input_tokens: 0.20
            output_tokens: 0.60

Selects the eligible backend with the lowest estimated request cost: input estimate x input rate + max_tokens x output rate, computed from the per-model PricingInfo in USD per 1M tokens.

Pricing source. The backend's model_configs[].metadata.pricing comes first, then the global model-metadata catalog. Provider-declared prices win over catalog values, so replicas of one model differ in price only when each declares its own entry. A backend-local model_configs entry that carries a metadata block but states no price deliberately shadows the catalog, so it is treated as unpriced rather than inheriting a shared rate; an entry without a metadata block falls through to the catalog's price.

Unpriced candidates. A candidate whose model has no pricing metadata ranks last, never as a free candidate. When no candidate in the pass is priced, the strategy behaves exactly like RoundRobin, so it never silently pins traffic to an unpriced backend.

Ties. Equal estimated costs break on the output rate and then rotate through the pool counter, so equal-priced replicas share traffic instead of pinning the first one.

Request sizing. Dispatch paths that hold the parsed request body price it from its tokenizer-free input estimate plus the request's completion ceiling: max_tokens or max_completion_tokens on chat, max_output_tokens on Responses, and max_tokens on Anthropic Messages.

Paths that reach selection without a parsed body use a rate-only basis: one million input tokens and no completion, which ranks by input rate and breaks ties on output rate. That basis exceeds the threshold of typical long-context tiers, so on these paths the long-context rate applies when the model declares one. Audio, tokenize, realtime, and ACP selection reach the seam this way and therefore rank on that basis rather than on the request's actual prompt size.

Validation. config validate warns at selection_strategy when a model served by two or more enabled backends has fewer than two priced replicas, because those replicas cannot be told apart by price. Only models declared through models or model_configs are examined; discovery-only fleets are not warned about a condition the configuration cannot state.

Composition. Smart-routing tiers still narrow the candidate set before the cost ranking runs. Hub-directed price arbitrage (src/control_plane/arbitrage.rs) rewrites a request to the cheapest healthy member of a Hub equivalence class before local selection, so it keeps precedence when both apply.

Scorer precedence¶

BackendPool evaluates registered scorers before falling back to selection_strategy. When the KV cache index is enabled and finds a candidate whose cached prefix overlap clears the configured threshold, the scorer selects that backend directly; otherwise selection_strategy runs. As a result, traffic distribution when scorers are active will intentionally differ from pure round-robin or weighted ratios.

See KV Cache Architecture for scorer configuration and thresholds.

Engine-load-aware selection¶

Setting routing.engine_load.enabled: true adds an engine-load term to the KV overlap scorer, fed by the engine statistics poller. Among backends that hold cached KV blocks for a request's prefix, it favors candidates reporting fewer waiting_requests and lower kv_cache_usage. Unlike router-local in-flight counters, this term accounts for requests queued directly on the inference engine and KV cache pressure approaching preemption.

Comparison: EngineLoad strategy vs. KV scorer engine-load term¶

Operating bounds and guards¶

• Queue normalization: waiting_requests is normalized by total_slots when every fresh candidate reports total_slots, and otherwise by the candidate set's maximum waiting_requests so mixed vLLM/llama.cpp pools are compared on a consistent scale.
• Staleness guard (max_staleness_intervals): Snapshots older than engine_stats.interval * max_staleness_intervals (default 3 intervals) are ignored. Candidates without a fresh snapshot retain their base overlap score.
• Hysteresis dead band (balance_abs_threshold, balance_rel_threshold): The engine-load term activates only when the spread in fresh waiting_requests across the live candidate set exceeds both balance_abs_threshold (default 64) and balance_rel_threshold (default 1.5). Note that the gate opens strictly on waiting_requests spread; differences in kv_cache_usage alone do not open the gate, though kv_cache_usage contributes to the load score once the gate is open.
• Trust and influence bounds: Because waiting_requests and kv_cache_usage (clamped to [0.0, 1.0]) are self-reported by each backend, three constraints bound their impact:
  1. Only candidates that already cleared min_overlap_threshold for the request's prefix are eligible.
  2. The additive weight is capped by engine_load_weight (default 0.3 on top of base weights summing to 1.0), so self-reported load alone cannot override a large overlap, in-flight load, or health gap.
  3. Inside the hysteresis dead band, the term contributes zero.

Saturation admission gate (routing.engine_load.admission)¶

When routing.engine_load.admission.enabled: true is configured, the router checks candidate KV saturation before running any selection strategy:

• If every healthy candidate reports a fresh kv_cache_usage above admission.kv_usage_threshold (default 0.98), selection is rejected (reason="admission_reject") instead of queueing onto an engine about to preempt.
• On /v1/chat/completions, this rejection returns a retryable HTTP 503 that triggers fallback.fallback_chains under the BackendUnhealthy condition.
• Any candidate without a fresh snapshot is treated as non-saturated, ensuring a single backend can trigger an admission reject only for requests where it is the sole healthy candidate.

Usage-weighted selection (routing.usage_weight)¶

routing.usage_weight ranks candidates by the share of a provider's declared token or request quota that the router has not spent yet. It is an additive scoring term, not a selection_strategy value, so it composes with every strategy and with the KV overlap and engine-load terms.

Declare each backend's provider limit, then turn the term on:

backends:
  - name: provider-a
    url: https://api.provider-a.example/v1
    provider_tpm: 200000
    provider_rpm: 500
  - name: provider-b
    url: https://api.provider-b.example/v1
    provider_tpm: 40000

routing:
  usage_weight:
    enabled: true
    weight: 0.2
    staleness_interval: 60s

The declaration states what the provider allows. It enforces nothing: a backend that exceeds provider_tpm is still dispatched until the provider refuses it. Use rate_limiting or per-key token budgets for enforcement.

Ranking contract¶

• Declared candidate: scores weight x remaining_share, where remaining_share is (configured - consumed) / configured clamped to 0.0..=1.0. A candidate that declares both dimensions is limited by whichever is closest to exhaustion.
• Undeclared candidate: scores exactly 0.0 and keeps its pre-existing score. An unknown quota is never treated as unlimited and never as zero remaining.
• Exhausted candidate: a declared candidate with no remaining budget also scores 0.0, the same as an undeclared one. It is never pushed below neutral.
• Strict preference: the term selects a backend only when its maximum contribution is positive and its combined score is strictly greater than every other candidate's. A tie falls back to selection_strategy, so two identically budgeted replicas keep rotating instead of flapping.

With weight: 0.2 the term cannot clear the scorer threshold (0.3) on its own. It can still decide a pass when the KV or engine-load terms report a weak, sub-threshold preference: in that case the term breaks the near-tie in favor of the candidate with more remaining budget.

Data source and window¶

• Consumption comes from the router's own token metering, per backend, in a tumbling window of staleness_interval (1s to 1h, default 60s). The window restarts at zero, including after a router restart, so a backend that was already exhausted can be preferred for the first window after a restart.
• The RPM dimension counts completed upstream attempts that reported usage. Attempts that fail before reporting usage are not counted, so it approximates request volume rather than a provider token bucket.
• Provider response headers and engine statistics are later stages of the same design; neither is read here yet.

Operating notes¶

• Metric: every pass the term decides or cannot settle records routing_usage_weight_decisions_total{backend,reason} with reason in applied, no_declaration, or no_differentiation (see Metrics). A pass the KV overlap or engine-load terms already settle keeps its KV metrics and is not counted here.
• Validation: weight must be finite in 0.0..=1.0, staleness_interval must parse between 1s and 1h, and provider_tpm/provider_rpm must be at least 1. config validate warns when the term is enabled and no enabled backend declares a limit, and when a declaration sits under a disabled term.
• Reload: routing.usage_weight.* and the quota declarations apply on the next published reload, including registering or removing the term when enabled flips.
• Default: with the block absent or disabled, no scorer runs without a prefix key and selection is identical to a build without this section.

Candidate eligibility and health checks¶

Every selection strategy chooses strictly from the filtered candidate set for the request. Candidate filtering enforces:

• Model availability and alias resolution
• Backend health status
• Circuit-breaker and retry exclusions
• Per-API-key allowed_backends and model visibility rules
• Internal backend isolation (internal: true)
• AppProxy ROUTER published-model restrictions (when active)

Configure background health checks globally under health_checks:

health_checks:
  enabled: true
  interval: 30s
  timeout: 5s
  unhealthy_threshold: 3
  healthy_threshold: 2

Hot reload and validation¶

Both selection_strategy and prefix_routing.load_factor_epsilon support live hot reload. Updating either value in config.yaml swaps the active strategy or CHWBL load factor atomically without restarting the router or resetting backend statistics and in-flight counters.

Validate configuration changes before applying them with:

continuum-router config validate /etc/continuum-router/config.yaml

Monitoring¶

Query backend health and prefix-routing distribution via the Admin API:

curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8080/admin/backends

curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8080/admin/prefix-routing/stats

When Prometheus metrics are enabled, prefix routing exports:

• continuum_prefix_routing_requests_total{strategy=...}
• continuum_prefix_routing_backend_distribution{backend=...}
• continuum_prefix_routing_prefix_cardinality

Choosing a strategy¶

routing.usage_weight is deliberately not in this table: it is an additive term over whichever strategy is chosen, not a strategy value. See Usage-weighted selection.

Upstream-observed quota¶

Upstream providers report their own quota headroom on every response. Router-local accounting (routing.usage_weight, above) ranks on the declared provider_tpm/provider_rpm minus what the router itself has metered, and routing.quota_aware (#2318) adds the observed half on top: an upstream that is about to 429 is deprioritized before the client sees the error.

1. Header capture: the router parses the provider's documented per-dimension rate-limit headers and stores the freshest snapshot per backend. The parsed set is Anthropic anthropic-ratelimit-requests-{limit,remaining}, anthropic-ratelimit-tokens-{limit,remaining}, anthropic-ratelimit-input-tokens-{limit,remaining}, and anthropic-ratelimit-output-tokens-{limit,remaining}; the anthropic-ratelimit-unified-{requests,tokens}-* aliases the Anthropic ingress forwards (is_forwardable_upstream_response_header, #1687); and OpenAI x-ratelimit-limit-requests / x-ratelimit-remaining-requests plus x-ratelimit-limit-tokens / x-ratelimit-remaining-tokens.
2. Normalization: a dimension ranks only when both the remaining value and its matching -limit header are present and the limit is positive, as remaining / limit clamped to 0.0..=1.0. A remaining value without a denominator is exposed on the Admin API but never ranked, so the router-local meter stays the fallback instead of being outranked by an invented ratio. The request axis takes the provider's request share; the token axis takes the smallest usable share across tokens, input-tokens, and output-tokens.
3. Selection: a fresh observation replaces the router-local estimate per axis, so a backend with observed data participates without a declared provider_tpm / provider_rpm. An absent or expired snapshot falls back to the declared estimate, and with the block disabled selection is byte-identical to a build without it. This stays an additive term over whichever selection_strategy is chosen; there is no new strategy value.
4. Visibility: GET /admin/stats/backends attaches a quota object beside engine_stats for each backend, and GET /admin/backends/{name}/engine-stats returns it too. It carries the observation time, age against the live TTL, each captured dimension with its ratio, and the recognized source headers. Selection inputs are counted in routing_quota_aware_decisions_total{backend,reason} with reason in observed_used, stale, absent, unusable.

Capture happens at every request-serving upstream send: the shared HTTP send (including the OAuth 401 retry), the Unix-socket transport, the streaming handler, native and converted Anthropic sends (Messages, web search, count_tokens, and the tokenizer's engine call), Responses sends (native streaming, pass-through, and lifecycle), the responses-only bridge and shadow attempts, audio, image generation and edit, and Gemini embedding. Only a resolved response records an observation; a transport error, connect failure, or timeout records nothing. Bedrock-runtime and self-hosted engines return no comparable headers, so they never produce an observation, and management calls such as fine-tuning or model discovery are not routed traffic and are not captured.

The headers describe the provider key, not the router backend. One snapshot is kept per backend and each response replaces it, so observations from different credentials never mix and the snapshot is meaningful for single-credential backends. A backend whose URL changes, or one removed from the configuration, drops its snapshot when the change is applied, so a same-name replacement never inherits another endpoint's headroom.

Unlike the engine-stats scorer, quota headroom comes from the response path rather than a poll loop, so the snapshot is as fresh as the last request to that backend and needs no separate fetch. Configure it under routing.quota_aware in the configuration guide; it is inert unless routing.usage_weight.enabled is also true.

See also¶

• Configuration Guide
• Performance Guide
• Metrics Guide
• KV Cache Architecture