Performance Guide¶

This guide describes the implemented performance controls and a measurement-first tuning workflow. End-to-end latency and throughput depend primarily on the selected model, provider, network, request size, streaming behavior, enabled middleware, and host limits; the project does not promise a universal requests-per-second or memory target.

Measure before tuning¶

The repository contains Criterion benchmarks under benches/:

cargo bench
cargo bench --bench sse_pipeline_micro

Run benchmarks on the same target and feature set used in production. For HTTP load tests, use a real request body, authentication mode, and backend mix; a /health benchmark does not represent proxy traffic.

Example with wrk and a POST script:

-- scripts/chat_completion.lua
wrk.method = "POST"
wrk.body = '{"model":"gpt-5.4-nano","messages":[{"role":"user","content":"Hello"}]}'
wrk.headers["Content-Type"] = "application/json"

wrk -t12 -c400 -d30s --latency \
  -s scripts/chat_completion.lua \
  http://localhost:8080/v1/chat/completions

Test the backend directly as a baseline. The difference between direct and routed traffic is more meaningful than results copied from another host.

SSE pipeline benchmark¶

All /v1/chat/completions SSE responses use the shared build_sse_pipeline adapters for chunk deadlines, UTF-8 boundary handling, SSE parsing, reasoning normalization, error conversion, keep-alive injection, and serialization.

A repository benchmark on an Apple M1 Ultra (macOS 26.5, rustc 1.95, one benchmark thread) measured:

Treat these as one-host microbenchmark results, not a service-level guarantee. Re-run the benchmark for the release, compiler, architecture, and feature set you deploy.

Application controls¶

Connection pool¶

server.connection_pool_size sets the maximum idle HTTP connections kept per backend. Increase it only when measurements show connection churn or insufficient reuse; each additional idle connection consumes resources.

server:
  connection_pool_size: 100

The equivalent one-off CLI override is:

continuum-router --config config.yaml --connection-pool-size 100

The router pre-warms backend connections at startup. Reusing the shared HTTP client reduces handshake overhead, but the observed benefit depends on provider and network behavior.

Backend selection¶

Use the top-level selection_strategy field. Do not use routing.strategy; it is not the production backend-selection control.

selection_strategy: LeastLatency

backends:
  - name: primary
    url: http://backend-1:8000
    weight: 3
  - name: secondary
    url: http://backend-2:8000
    weight: 1

Available values are:

• RoundRobin
• WeightedRoundRobin
• LeastLatency
• Random
• ConsistentHash
• PrefixAwareHash

LeastLatency uses observed backend latency. WeightedRoundRobin uses each backend's weight. ConsistentHash provides stable affinity, while PrefixAwareHash uses prompt-prefix affinity and the prefix_routing settings to improve distributed KV-cache reuse.

Timeouts¶

Use shorter timeouts only when abandoning slow provider work is preferable to waiting for it. Keep the client's total deadline longer than the router's retry and request budget.

timeouts:
  connection: 10s
  request:
    standard:
      first_byte: 30s
      total: 180s
    streaming:
      first_byte: 60s
      chunk_interval: 30s
      total: 600s

Retries¶

Retries improve transient-failure recovery but add work and tail latency. Low-latency deployments may use fewer attempts after validating provider reliability.

retry:
  max_attempts: 2
  initial_delay: 100ms
  max_delay: 2s
  backoff_multiplier: 2.0
  jitter: true
  retryable_status_codes: [429, 502, 503, 504]
  retryable_errors: [ConnectionError, TimeoutError]
  timeout: 10s

Health checks¶

Frequent checks detect failures sooner but increase periodic provider traffic:

health_checks:
  enabled: true
  interval: 30s
  timeout: 5s
  unhealthy_threshold: 3
  healthy_threshold: 2

Response cache¶

The optional response cache serves eligible deterministic requests and can reduce backend calls. Size it based on measured entry size and hit rate.

response_cache:
  enabled: true
  backend: memory
  capacity: 1000
  ttl: 5m
  max_response_size: 1048576

Redis and tiered backends require their corresponding build features and configuration. See Advanced Configuration before enabling them.

Model-list cache¶

Model aggregation uses a 60-second hard TTL, an 80% soft TTL, stale-while-revalidate, request coalescing, a 10-second background check interval, and bounded empty-result backoff. These internal values are not configurable. POST /v1/models/refresh performs a rate-limited immediate refresh.

Example profiles¶

These are starting points, not universal optimums. Validate every profile with continuum-router config validate and a representative load test.

Lower tail latency¶

selection_strategy: LeastLatency

server:
  bind_address: 0.0.0.0:8080
  connection_pool_size: 100

retry:
  max_attempts: 1
  initial_delay: 100ms
  max_delay: 1s
  backoff_multiplier: 2.0
  jitter: true
  retryable_status_codes: [429, 502, 503, 504]
  retryable_errors: [ConnectionError, TimeoutError]
  timeout: 5s

health_checks:
  enabled: true
  interval: 10s
  timeout: 5s
  unhealthy_threshold: 3
  healthy_threshold: 2

Smaller idle footprint¶

server:
  connection_pool_size: 10

response_cache:
  enabled: false

logging:
  level: warn
  format: json

Higher concurrency¶

server:
  connection_pool_size: 500
  # Bound residency as well as arrival. Size against the container memory
  # limit: see Capacity planning in the deployment guide.
  max_concurrent_requests: 256

rate_limiting:
  enabled: true
  storage: memory

Complete the rate-limit policy rather than copying a partial block into production; see Rate Limiting. connection_pool_size is the outbound per-backend pool and does not bound inbound concurrency; max_concurrent_requests does, and over the ceiling the router sheds with 503 plus a Retry-After instead of queueing. Raise the container memory limit in the same change, since the two are one decision.

Prometheus monitoring¶

When compiled with metrics and enabled at runtime, Prometheus metrics are exposed at the configured endpoint:

metrics:
  enabled: true
  endpoint: /metrics
  auth:
    enabled: true
    username: metrics
    password: "${METRICS_PASSWORD}"

Useful metrics include:

• http_requests_total
• http_request_duration_seconds
• backend_healthy
• backend_current_load
• http_active_connections

Check the running /metrics output before writing dashboards because feature-gated subsystems add metrics only when compiled and active.

Example queries:

rate(http_requests_total[5m])
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
backend_healthy
backend_current_load

Operating-system limits¶

For high concurrency, inspect file descriptor and listen-backlog limits before changing them:

ulimit -n
sysctl net.core.somaxconn
ss -s

Apply host-level sysctl or security-limit changes only after testing them for the operating system, kernel, container runtime, and traffic pattern. The project does not require one universal kernel tuning profile.

Diagnosing bottlenecks¶

High latency¶

1. Compare the same request directly against the selected backend.
2. Check /admin/health and /admin/circuit/all with Admin credentials.
3. Inspect http_request_duration_seconds and backend latency/load metrics.
4. Verify that retry and timeout budgets do not multiply tail latency.
5. Temporarily enable targeted RUST_LOG=continuum_router=debug logging; avoid logging secrets.

High memory¶

1. Measure resident memory while reproducing the workload.
2. Read the in-flight request count first, not the request rate. Worst-case resident memory is concurrency times the per-request budget, and a rate limit bounds arrivals per unit time rather than the number resident at once: at a 30 second mean duration, 10 requests per second is roughly 300 in flight. active_requests in the control-plane supply report and the in-flight gauges are the numbers to look at.
3. Set server.max_concurrent_requests if it is unset. Without it nothing bounds the concurrency factor, so the per-request budgets do not imply a process bound at all. See Capacity planning: memory for the per-request terms and the arithmetic.
4. Inspect request/response sizes, concurrent streams, and response-cache capacity. Files API transfers stream to disk and cost a 64 KiB buffer each regardless of files.max_file_size, so the largest single memory term is the per-request resolved file content (32 MiB raw, about 43 MiB base64-expanded).
5. Reduce response_cache.capacity or disable the response cache to compare.
6. Profile the release binary with a platform-appropriate allocator/profiler.

The model-list cache capacity is internal and not configurable.

Low throughput¶

1. Compare provider throughput directly.
2. Inspect rate-limit rejections, circuit state, backend health, and in-flight load.
3. Check connection reuse and file descriptor limits before increasing connection_pool_size.
4. Test streaming and non-streaming workloads separately.
5. Scale horizontally only after confirming the shared state/storage choices are appropriate for multiple instances.

See also¶

• Configuration Guide
• Deployment Guide
• Metrics Guide
• Load Balancing Guide