Load Balancing Guide¶

Continuum Router first narrows a request to backends that are live, healthy, visible to the caller, and capable of serving the resolved model. It then applies scorer-based routing when configured and otherwise uses the top-level selection_strategy.

Canonical configuration¶

selection_strategy: RoundRobin

backends:
  - name: gpu-a
    url: http://gpu-a:8000
    weight: 3
    models: [llama-3.3-70b]
  - name: gpu-b
    url: http://gpu-b:8000
    weight: 1
    models: [llama-3.3-70b]

Use selection_strategy, backends[].weight, backends[].models, smart_routing, and prefix_routing to control selection. The accepted routing: section has no runtime consumer and must not be used for production behavior.

The CLI can override the startup strategy:

continuum-router --config config.yaml --selection-strategy LeastLatency

CONTINUUM_SELECTION_STRATEGY is a direct startup environment override, parsed with the same rules as --selection-strategy (all six strategy names, case-insensitive, with snake_case and kebab-case spellings accepted too):

export CONTINUUM_SELECTION_STRATEGY=LeastLatency
continuum-router --config config.yaml

Precedence, highest to lowest: --selection-strategy > CONTINUUM_SELECTION_STRATEGY > the file's selection_strategy value > the built-in RoundRobin default. An unrecognized value in the environment variable fails startup with a clear error instead of silently falling back to the file value or the default.

Strategies¶

RoundRobin (default)¶

selection_strategy: RoundRobin

Walks the eligible backend list using an atomic counter. It is a good default for comparable stateless backends. It does not use backend weights or observed latency.

WeightedRoundRobin¶

selection_strategy: WeightedRoundRobin

backends:
  - name: large
    url: http://large:8000
    weight: 3
  - name: small
    url: http://small:8000
    weight: 1

Selects eligible backends in proportion to their positive weights over time. The current implementation samples a weighted target for each request rather than enforcing a deterministic 3,3,3,1 sequence. If all eligible weights are zero, it falls back to round-robin behavior.

LeastLatency¶

selection_strategy: LeastLatency

Chooses the eligible backend with the lowest recorded average response time. Until a candidate has observations, selection falls back through the round-robin counter. This strategy optimizes observed latency; it does not predict request complexity and may concentrate traffic on one candidate.

Random¶

selection_strategy: Random

Chooses one eligible backend uniformly for each request. Short windows can be uneven even when the long-run distribution is balanced.

ConsistentHash¶

selection_strategy: ConsistentHash

Hashes the resolved model name onto a backend hash ring. Requests for the same model and candidate set normally select the same backend. This is model affinity, not user/session affinity, and it can be imbalanced when only a few model keys dominate traffic.

PrefixAwareHash¶

selection_strategy: PrefixAwareHash

prefix_routing:
  enabled: true
  max_prefix_length: 1024
  load_factor_epsilon: 0.25

When prefix extraction is enabled, the router derives a SHA-256 key from the model plus the system prompt, or from the model plus the first user message when no system prompt is present. It routes matching prefixes to the same backend and uses consistent hashing with bounded loads (CHWBL) to overflow a busy preferred backend. When no prefix key is available, it falls back to model-based consistent hashing.

The bounded-load cap is:

ceil((total_in_flight + 1) * (1 + epsilon) / backend_count)

Lower epsilon values favor balance; higher values preserve more affinity. Prefix extraction supports chat, Responses, and Anthropic-compatible typed paths where the code can obtain an eligible textual prefix.

prefix_routing.virtual_nodes sizes the consistent-hash ring (default 150 replicas per backend) and is applied to the live pool on hot-reload; changing it rebuilds the ring. anthropic_cache_control_injection (default false) is a fleet-wide master enable that forces automatic Anthropic cache_control injection on eligible native-Anthropic requests even where a backend set anthropic_auto_cache_control: false; it is read per-request and preserves user-supplied cache_control blocks.

Scorer precedence¶

BackendPool evaluates registered scorers before the configured fallback strategy. For example, when the KV cache index is enabled and has a sufficiently strong overlap score, it can select the backend that owns the relevant cached tokens; otherwise selection_strategy runs. This means observed distribution may intentionally differ from pure round-robin or weighted ratios.

See KV Cache Architecture for scorer configuration and thresholds.

Health and eligibility¶

Selection never intentionally chooses outside the already resolved candidate set. Candidate filtering includes:

• Model availability and alias resolution
• Backend health
• Circuit/retry exclusion
• Per-key allowed_backends and model visibility
• Internal backend filtering where applicable
• AppProxy ROUTER published-model restrictions when that feature is active

Configure health checks globally:

health_checks:
  enabled: true
  interval: 30s
  timeout: 5s
  unhealthy_threshold: 3
  healthy_threshold: 2

There is no dynamic_weight_adjustment, health_score_threshold, Custom, or Geographic selection configuration in the current schema.

Changing strategy¶

Editing selection_strategy is a live hot-reload. The configuration watcher swaps the new strategy onto the running backend pool atomically, so the next selection uses it without a restart. Backend membership, per-backend statistics, and in-flight request accounting are preserved across the swap; only the selection algorithm changes.

The pool's prefix_routing.load_factor_epsilon (the CHWBL load factor) reloads the same way: the new value is swapped onto the live pool without a restart.

You can still validate the file before applying a change:

continuum-router config validate /etc/continuum-router/config.yaml

Monitoring¶

With Admin authentication as configured:

curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8080/admin/backends

curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8080/admin/prefix-routing/stats

/admin/backends reports configured backend health information. /admin/prefix-routing/stats reports prefix decision counters and live in-flight distribution.

When metrics are compiled and enabled, prefix routing exports:

• continuum_prefix_routing_requests_total{strategy=...}
• continuum_prefix_routing_backend_distribution{backend=...}
• continuum_prefix_routing_prefix_cardinality

Inspect the running /metrics output before depending on a series because feature-gated metrics exist only in builds that register them.

Choosing a strategy¶

Always test with the real eligible model set. If a model exists on only one backend, no selection strategy can distribute that model elsewhere.

See also¶

• Configuration Guide
• Performance Guide
• Metrics Guide
• KV Cache Architecture