Load Balancing Guide¶

Continuum Router first narrows a request to backends that are live, healthy, visible to the caller, and capable of serving the resolved model. It then applies scorer-based routing when configured and otherwise uses the top-level selection_strategy.

Canonical configuration¶

selection_strategy: RoundRobin

backends:
  - name: gpu-a
    url: http://gpu-a:8000
    weight: 3
    models: [llama-3.3-70b]
  - name: gpu-b
    url: http://gpu-b:8000
    weight: 1
    models: [llama-3.3-70b]

Use selection_strategy, backends[].weight, backends[].models, smart_routing, and prefix_routing to control selection. The accepted routing: section has no runtime consumer and must not be used for production behavior.

The CLI can override the startup strategy:

continuum-router --config config.yaml --selection-strategy LeastLatency

CONTINUUM_SELECTION_STRATEGY is a direct startup environment override, parsed with the same rules as --selection-strategy (all seven strategy names, case-insensitive, with snake_case and kebab-case spellings accepted too):

export CONTINUUM_SELECTION_STRATEGY=LeastLatency
continuum-router --config config.yaml

Precedence, highest to lowest: --selection-strategy > CONTINUUM_SELECTION_STRATEGY > the file's selection_strategy value > the built-in RoundRobin default. An unrecognized value in the environment variable fails startup with a clear error instead of silently falling back to the file value or the default.

Strategies¶

RoundRobin (default)¶

selection_strategy: RoundRobin

Walks the eligible backend list using an atomic counter. It is a good default for comparable stateless backends. It does not use backend weights or observed latency.

WeightedRoundRobin¶

selection_strategy: WeightedRoundRobin

backends:
  - name: large
    url: http://large:8000
    weight: 3
  - name: small
    url: http://small:8000
    weight: 1

Selects eligible backends in proportion to their positive weights over time. The current implementation samples a weighted target for each request rather than enforcing a deterministic 3,3,3,1 sequence. If all eligible weights are zero, it falls back to round-robin behavior.

LeastLatency¶

selection_strategy: LeastLatency

Chooses the eligible backend with the lowest recorded average response time. Until a candidate has observations, selection falls back through the round-robin counter. This strategy optimizes observed latency; it does not predict request complexity and may concentrate traffic on one candidate.

Random¶

selection_strategy: Random

Chooses one eligible backend uniformly for each request. Short windows can be uneven even when the long-run distribution is balanced.

ConsistentHash¶

selection_strategy: ConsistentHash

Hashes the resolved model name onto a backend hash ring. Requests for the same model and candidate set normally select the same backend. This is model affinity, not user/session affinity, and it can be imbalanced when only a few model keys dominate traffic.

PrefixAwareHash¶

selection_strategy: PrefixAwareHash

prefix_routing:
  enabled: true
  max_prefix_length: 1024
  load_factor_epsilon: 0.25

When prefix extraction is enabled, the router derives a SHA-256 key from the model plus the system prompt, or from the model plus the first user message when no system prompt is present. It routes matching prefixes to the same backend and uses consistent hashing with bounded loads (CHWBL) to overflow a busy preferred backend. When no prefix key is available, it falls back to model-based consistent hashing, and that fallback ring walk carries no bounded-load cap: it never overflows, so every request for one model lands on one backend. PrefixAwareHash therefore requires prefix_routing.enabled: true; with prefix routing off, every request takes that uncapped fallback, and config validate and the router logs report the combination as a warning.

The bounded-load cap is:

ceil((total_in_flight + 1) * (1 + epsilon) / backend_count)

Lower epsilon values favor balance; higher values preserve more affinity. Prefix extraction supports chat, Responses, and Anthropic-compatible typed paths where the code can obtain an eligible textual prefix.

prefix_routing.virtual_nodes sizes the consistent-hash ring (default 150 replicas per backend) and is applied to the live pool on hot-reload; changing it rebuilds the ring. anthropic_cache_control_injection (default false) is a fleet-wide master enable that forces automatic Anthropic cache_control injection on eligible native-Anthropic requests even where a backend set anthropic_auto_cache_control: false; it is read per-request and preserves user-supplied cache_control blocks.

EngineLoad¶

selection_strategy: EngineLoad

engine_stats:
  enabled: true

routing:
  engine_load:
    balance_abs_threshold: 64
    balance_rel_threshold: 1.5
    base_strategy: RoundRobin

Routes to the candidate whose engine reports the lightest load, reading the same engine statistics as the scorer term described under Engine-load-aware selection and applying the identical comparison: waiting_requests normalized by the pass-wide basis, plus kv_cache_usage. The difference is scope. The scorer term ranks only a request's prefix holders and runs only when the KV overlap scorer is registered; this strategy ranks the whole eligible candidate set and needs neither the KV index nor a prefix key, which is what makes the engine statistics usable on a fleet without prefix routing.

One selection reaches exactly one of three outcomes:

• Ranked (reason="strategy_engine_load"): every eligible candidate has a fresh snapshot stating waiting_requests, and the spread clears both balance thresholds. The least-loaded candidate wins.
• Stale fallback (reason="strategy_stale_fallback"): at least one eligible candidate makes no fresh statement, so the comparison is undefined and routing.engine_load.base_strategy decides. Requiring a statement from every candidate is deliberate: ranking a subset would route every request during an imbalance to whichever backends happen to publish statistics and starve the ones that do not, which is the wrong answer on a mixed self-hosted and hosted-API fleet.
• Hysteresis hold (reason="strategy_hysteresis_hold"): every candidate states a fresh queue but the spread sits inside the dead band, so the base strategy decides. This is the steady-state answer on a balanced fleet.

The strategy is therefore a corrective shove during genuine imbalance layered on top of base_strategy (default RoundRobin; use WeightedRoundRobin on a heterogeneous fleet so the balanced case still honors per-backend weights). That layering is also what bounds herding: a snapshot refreshes only once per engine_stats.interval, so a ranked pass sends the whole interval's traffic to one backend, and the dead band is what keeps ranking from engaging until the imbalance is large enough to be worth that.

selection_strategy: EngineLoad does not read routing.engine_load.enabled, which gates the scorer term only; naming the strategy is itself the opt-in. It does need engine_stats.enabled: true for data, and config validate warns at the selection_strategy path when that is missing. routing.engine_load.base_strategy may not be EngineLoad, which is rejected at load. The staleness bound (engine_stats.interval * routing.engine_load.max_staleness_intervals) and the saturation admission hint apply exactly as they do for the scorer term; the hint runs at the shared selection seam ahead of every strategy, so it needs no separate configuration here.

Scorer precedence¶

BackendPool evaluates registered scorers before the configured fallback strategy. For example, when the KV cache index is enabled and has a sufficiently strong overlap score, it can select the backend that owns the relevant cached tokens; otherwise selection_strategy runs. This means observed distribution may intentionally differ from pure round-robin or weighted ratios.

See KV Cache Architecture for scorer configuration and thresholds.

Engine-load-aware selection¶

With routing.engine_load.enabled: true, the KV overlap scorer gains an additive engine-load term fed by the engine statistics poller: among the backends holding cached data for a request's prefix, it prefers the one whose engine reports fewer waiting_requests (normalized by total_slots when every fresh candidate states one, otherwise by the request candidate set's maximum waiting_requests) and lower kv_cache_usage. Router-local in-flight counters cannot see requests queued inside an engine or a KV cache about to force preemption; this term can.

Three guards bound the term:

• Staleness: a snapshot older than engine_stats.interval * routing.engine_load.max_staleness_intervals (default 3 intervals) is treated as absent. A candidate without a fresh snapshot keeps its base score, and when no candidate has one the whole pass matches the base scoring exactly.
• Hysteresis: the term ranks candidates only while the fresh waiting_requests spread exceeds BOTH balance_abs_threshold (default 64) and balance_rel_threshold (default 1.5), mirroring the SGLang Model Gateway's balance gates. Inside that dead band the term contributes nothing, so two backends whose queues oscillate around each other cannot flap selection. The dead band is evaluated on the waiting_requests spread alone; kv_cache_usage cannot by itself open the gate, so a fleet with balanced or zero queues but very different KV pressure gets no engine-load ranking at all, even though kv_cache_usage is part of the load score once the gate has tripped.
• Scope: the term ranks only among prefix holders, like the load and health terms, and it requires the scorer to exist at all (prefix_routing.enabled plus an enabled kv_cache_index) and engine_stats.enabled: true for data. config validate warns when either dependency is missing. The gate, normalization, cached result, and routing_engine_load_decisions_total backend label are all bound to the exact live candidate set for the selection pass. Model visibility, health, circuit/retry exclusions, and per-key permissions therefore prevent an unrelated pool member from opening the gate or leaking a cached preference into the request.

Trust model. waiting_requests and kv_cache_usage are numbers each engine reports about itself, so a compromised or buggy backend can understate its own load to attract traffic. Three properties bound what that buys it. The term is applied only to prefix holders that already cleared min_overlap_threshold, so a backend that holds nothing for the request cannot be pulled in at all. The term is capped at engine_load_weight (default 0.3) against base weights that sum to 1.0, so no self-reported figure can override a decisive overlap, load, or health difference on its own. And the balance dead band means the term is inert entirely while queues are close. Raising engine_load_weight toward 1.0 raises the ceiling accordingly. On the other side, the admission hint requires every healthy candidate to report fresh saturation, so a single engine can only refuse requests for which it is the sole candidate. kv_cache_usage is clamped to [0, 1] on ingestion and a non-finite reading is discarded, so an absurd value cannot escape that range.

Every scoring pass records routing_engine_load_decisions_total{backend,reason} with reason set to engine_load (the term actively ranked, backend names the engine-preferred candidate), stale_fallback (no fresh engine statement), or hysteresis_hold (fresh data inside the dead band).

Separately, routing.engine_load.admission.enabled: true arms a saturation admission hint at the shared selection seam: when every healthy candidate has a fresh snapshot with kv_cache_usage above admission.kv_usage_threshold (default 0.98), selection is refused instead of queueing onto an engine about to preempt (reason="admission_reject"). On the chat path the refusal surfaces as a retryable 503 that participates in fallback.fallback_chains through the BackendUnhealthy trigger; a candidate with no fresh statement always admits, so the gate can only trip on engine truth. See the routing section of config.yaml.example for the full field reference and reload classes.

Health and eligibility¶

Selection never intentionally chooses outside the already resolved candidate set. Candidate filtering includes:

• Model availability and alias resolution
• Backend health
• Circuit/retry exclusion
• Per-key allowed_backends and model visibility
• Internal backend filtering where applicable
• AppProxy ROUTER published-model restrictions when that feature is active

Configure health checks globally:

health_checks:
  enabled: true
  interval: 30s
  timeout: 5s
  unhealthy_threshold: 3
  healthy_threshold: 2

There is no dynamic_weight_adjustment, health_score_threshold, Custom, or Geographic selection configuration in the current schema.

Changing strategy¶

Editing selection_strategy is a live hot-reload. The configuration watcher swaps the new strategy onto the running backend pool atomically, so the next selection uses it without a restart. Backend membership, per-backend statistics, and in-flight request accounting are preserved across the swap; only the selection algorithm changes.

The pool's prefix_routing.load_factor_epsilon (the CHWBL load factor) reloads the same way: the new value is swapped onto the live pool without a restart.

You can still validate the file before applying a change:

continuum-router config validate /etc/continuum-router/config.yaml

Monitoring¶

With Admin authentication as configured:

curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8080/admin/backends

curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8080/admin/prefix-routing/stats

/admin/backends reports configured backend health information. /admin/prefix-routing/stats reports prefix decision counters and live in-flight distribution.

When metrics are compiled and enabled, prefix routing exports:

• continuum_prefix_routing_requests_total{strategy=...}
• continuum_prefix_routing_backend_distribution{backend=...}
• continuum_prefix_routing_prefix_cardinality

Inspect the running /metrics output before depending on a series because feature-gated metrics exist only in builds that register them.

Choosing a strategy¶

Always test with the real eligible model set. If a model exists on only one backend, no selection strategy can distribute that model elsewhere.

See also¶

• Configuration Guide
• Performance Guide
• Metrics Guide
• KV Cache Architecture