Health, Caching & Logging¶

Health Checks Section¶

Configures backend health monitoring:

health_checks:
  interval: "30s"                  # Check frequency
  timeout: "10s"                   # Request timeout
  unhealthy_threshold: 3           # Failures before marking unhealthy
  healthy_threshold: 2             # Successes before marking healthy
  endpoint: "/health"              # Endpoint to check (per-type defaults below)
  warmup_check_interval: "1s"      # Accelerated interval during warmup
  max_warmup_duration: "300s"      # Maximum warmup detection duration
  block_startup: true              # Gate listener binding on pre-warm and the initial health check

Health Check Process: 1. Router queries the health endpoint on each backend 2. Successful responses increment success counter 3. Failed responses increment failure counter 4. Backends marked unhealthy after reaching failure threshold 5. Backends marked healthy after reaching success threshold 6. Only healthy backends receive traffic

Accelerated Warmup Health Checks¶

The router supports accelerated health checks during backend warmup, which is particularly useful for backends like llama.cpp that return HTTP 503 while loading models.

Backend States:

Warmup Configuration:

How it works:

1. When a backend returns HTTP 503, it enters the warming_up state
2. Health checks switch to the accelerated interval (default: 1 second)
3. Once the backend returns HTTP 200, it becomes ready and returns to normal interval
4. If warmup exceeds max_warmup_duration, the backend is marked as unhealthy

This reduces model availability detection latency from up to 30 seconds (worst case) to approximately 1 second.

Startup Behavior¶

health_checks.block_startup (default: true) controls whether the router binds its listeners before or after the first backend health check completes.

/health reports router liveness only, never backend health, so it stays meaningful either way. Set block_startup: false when an external readiness probe enforces a short deadline, since a single unreachable or stalling backend can otherwise push the pre-bind window past that deadline. Every health-check request is bounded by timeout regardless of block_startup, and the pre-warm request itself carries a fixed 10s cap, so a stalling backend no longer holds the router's shared client timeout (the streaming total, 600s by default) against startup.

health_checks:
  block_startup: false             # Bind listeners first; converge health in the background

Per-Backend Health Check Configuration¶

Each backend type has sensible default health check endpoints. You can override these defaults with a custom health_check configuration per backend.

Default Health Check Endpoints by Backend Type:

Fallback Behavior:

When the primary health check endpoint returns HTTP 404, the router automatically tries the fallback endpoints in order. This ensures compatibility with backends that may not implement all standard endpoints.

Custom Health Check Configuration:

backends:
  - name: vllm-custom
    type: vllm
    url: http://localhost:8000
    models:
      - my-model
    health_check:
      endpoint: /custom-health          # Primary endpoint
      fallback_endpoints:               # Tried if primary returns 404
        - /health
        - /v1/models
      method: GET                       # HTTP method: GET, POST, or HEAD
      timeout: 10s                      # Override global health check timeout
      accept_status:                    # Status codes indicating healthy
        - 200
        - 204
      warmup_status:                    # Status codes indicating model loading
        - 503

Health Check Configuration Options:

Example: Anthropic-Style Health Check:

For backends that use POST requests or accept error codes as healthy indicators:

backends:
  - name: custom-api
    type: generic
    url: http://localhost:9000
    models:
      - custom-model
    health_check:
      endpoint: /api/v1/health
      method: POST
      body:
        check: true
      accept_status:
        - 200
        - 400    # Bad request means server is up
        - 401    # Unauthorized means server is up
        - 429    # Rate limited means server is up

Timeouts Section¶

Request handling timeouts live under timeouts:, with separate budgets for non-streaming, streaming, and image-generation requests, plus model-specific overrides:

timeouts:
  connection: "10s"                # TCP connection establishment timeout
  request:
    standard:
      first_byte: "30s"            # Time to receive first byte
      total: "180s"                # Total request timeout
    streaming:
      first_byte: "60s"            # Time to first SSE chunk
      chunk_interval: "30s"        # Max time between chunks
      total: "600s"                # Total streaming timeout
    model_overrides:
      gpt-5-latest:
        streaming:
          total: "1200s"           # Per-model override

Timeout Considerations:

• Long totals accommodate slow model inference
• Streaming requests may take longer than non-streaming
• Balance between user experience and resource usage

Retry Section¶

Global retry configuration for resilience:

retry:
  max_attempts: 3                  # Maximum retry attempts
  initial_delay: "100ms"           # Initial delay between retries
  max_delay: "30s"                 # Cap on retry delays
  backoff_multiplier: 2.0          # Exponential backoff multiplier
  jitter: true                     # Add random jitter
  retryable_status_codes: [429, 502, 503, 504]

Retry Strategy:

• Exponential backoff: delays increase exponentially (100ms, 200ms, 400ms...)
• Jitter: adds randomness to prevent thundering herd
• Max delay: prevents extremely long waits

Model Aggregation Section¶

The model_aggregation section configures the aggregated model-list cache, the per-backend model fetcher, deduplication, the force-refresh endpoints, and the background refresh task. Every field has a default, so the section is optional; a resolved snapshot is applied at startup. Changing it requires a restart.

model_aggregation:
  cache_ttl: "60s"                  # Hard TTL for an aggregated model list
  soft_ttl_ratio: 0.8               # Stale-but-usable fraction of cache_ttl, (0.0, 1.0]
  request_timeout: "5s"             # Per-backend model-list fetch timeout
  max_retries: 2                    # Fetch retries per backend
  retry_delay: "500ms"              # Delay between fetch retries
  max_models_per_backend: 1000      # Cap on models from any single backend (> 0)
  max_total_models: 5000            # Cap on total models across backends (> 0)
  max_response_size: 10485760       # Max backend response size in bytes (> 0)
  empty_response_base_ttl: "5s"     # Empty-response backoff base
  empty_response_max_ttl: "60s"     # Empty-response backoff cap (>= base)
  max_cache_entries: 100            # Max cache entries, LRU beyond this (> 0)
  deduplication_strategy: merge_backends  # first_wins | last_wins | merge_backends
  allow_force_refresh: true         # false => POST /v1/models/refresh returns 403
  background_refresh:
    enabled: true                   # Proactive pre-expiry cache refresh
    check_interval: "10s"           # Background refresh polling cadence

Cache Stampede Prevention¶

The router implements three strategies to prevent cache stampede (thundering herd problem):

1. Singleflight Pattern: Only one equivalent aggregation request runs at a time
2. Stale-While-Revalidate: Stale data remains usable while refresh runs (past the soft_ttl_ratio point of cache_ttl)
3. Background Refresh: When background_refresh.enabled, a check_interval loop refreshes entries after the soft TTL

Empty results use bounded exponential backoff between empty_response_base_ttl and empty_response_max_ttl. Use the rate-limited POST /v1/models/refresh endpoint for an immediate refresh; when allow_force_refresh: false, that endpoint and the Admin refresh endpoint return the documented disabled (403) response.

Response Cache Section¶

Caches complete LLM responses for deterministic (temperature == 0) requests, both non-streaming and streaming. Repeated identical requests are served from memory without calling the backend.

response_cache:
  enabled: true                     # Enable response caching (default: false)
  backend: memory                   # Cache backend: "memory" (default), "redis", or "tiered"
  capacity: 1000                    # Maximum number of cached responses (LRU eviction)
  ttl: "5m"                         # Time-to-live for cached entries (e.g., "5m", "1h")
  max_response_size: 1048576        # Maximum response body size in bytes (default: 1 MiB)
  max_stream_buffer_size: 10485760  # Maximum streaming buffer size in bytes (default: 10 MiB)

Cache Eligibility¶

A response is cached only when all of the following conditions hold:

• response_cache.enabled is true
• The request's temperature field is 0 or absent (deterministic output)
• The response body does not exceed max_response_size
• For non-streaming requests: the response does not contain finish_reason: "error" in any choice
• For streaming requests: the stream completes successfully (receives the final [DONE] event); interrupted or errored streams are discarded and not cached

Non-eligible requests are forwarded to the backend normally.

Streaming Cache Behaviour¶

When caching is enabled and temperature is 0, streaming (stream: true) chat completions are also eligible for caching:

1. SSE events are buffered in memory alongside normal stream forwarding to the client; there is no observable latency increase.
2. On successful stream completion (final [DONE] event), the router reconstructs a complete chat.completion JSON object from the buffered chunks and stores it in the cache.
3. If the stream is interrupted (client disconnect, timeout, backend error), the buffer is discarded and nothing is stored.
4. On cache hit, the cached response is replayed to the client as a synthetic SSE stream (text/event-stream) with an X-Cache: HIT header.
5. The max_stream_buffer_size setting limits the streaming buffer. Streams whose total buffered content exceeds this threshold are not cached.

X-Cache Response Header¶

Every chat completion response includes an X-Cache header:

Cache Configuration Options¶

Redis/Valkey Backend¶

By default the response cache stores entries in process memory. To share the cache across multiple router instances or survive restarts, configure a Redis or Valkey backend.

Build requirement: The binary must be compiled with the redis-cache Cargo feature:

cargo build --release --features redis-cache

Configuration:

response_cache:
  enabled: true
  backend: redis                     # Select Redis backend
  ttl: "5m"
  redis:
    url: "redis://localhost:6379"    # plain TCP
    # url: "rediss://redis.example.com:6380"   # TLS
    # url: "redis://:password@localhost:6379"  # with auth
    pool_size: 8                     # connection pool size (default: 8)
    key_prefix: "cr:resp:"           # key namespace prefix (default: "cr:resp:")
    connect_timeout_ms: 3000         # connection timeout (default: 3000)
    command_timeout_ms: 1000         # per-command timeout (default: 1000)
    tls: false                       # use TLS (default: false)
    fallback_to_memory: true         # fallback to in-memory on failure (default: true)

Automatic failover: When fallback_to_memory is true (the default) and Redis becomes unreachable, the router transparently falls back to an in-memory cache and logs a warning. A background health monitor periodically sends PING commands and switches back to Redis once connectivity is restored. Set fallback_to_memory: false to disable this behaviour -- operations will fail instead of falling back.

Key safety: clear() uses SCAN + DEL with the configured prefix pattern instead of FLUSHDB. Only keys matching the prefix are removed, making it safe to share a Redis instance with other applications.

S3-Compatible Tiered Backend¶

The tiered backend composes a fast, bounded L1 cache with a virtually unlimited L2 cache backed by an S3-compatible API. Large responses are stored only in L2 to avoid eviction pressure on L1. When an L2 hit occurs, the value is optionally promoted back to L1 for faster subsequent access.

Any S3-compatible storage can be used as L2, including AWS S3, MinIO, Ceph, or VAST Data.

Build requirement: The binary must be compiled with the s3-cache Cargo feature:

cargo build --release --features s3-cache

Tiered cache write path:

1. Every set() writes to L2 (S3) unconditionally.
2. If the value is smaller than l1.max_value_size, it is also written to L1.
3. Values exceeding the threshold are demoted to L2-only; the demotion counter is incremented.

Tiered cache read path:

1. L1 is checked first. On hit, the value is returned immediately.
2. On L1 miss, L2 (S3) is queried.
3. On L2 hit, if tiered.promote_on_hit is true and the value fits in L1, it is promoted back to L1 with tiered.l1_promotion_ttl as the TTL.

TTL enforcement: Each S3 object carries an expires-at metadata field (Unix timestamp). On GET, if the timestamp has passed, the object is deleted lazily and a cache miss is returned. Use l2.ttl_override to keep L2 entries alive longer than the global ttl.

Configuration:

response_cache:
  enabled: true
  backend: tiered                    # Select tiered L1/L2 backend
  ttl: "5m"                          # Default TTL for both tiers

  l1:                                # L1 (hot) tier — fast, bounded
    type: memory                     # "memory" or "redis" (redis requires redis-cache feature)
    max_value_size: 1048576          # Values larger than this go to L2 only (default: 1 MiB)

  l2:                                # L2 (warm) tier — S3-compatible storage
    type: s3
    endpoint: "https://s3.example.com:8080"
    bucket: "llm-response-cache"
    key_prefix: "response-cache/"    # Object key prefix (default: "response-cache/")
    region: "us-east-1"              # Region string (default: "us-east-1")
    access_key: "${S3_ACCESS_KEY}"   # Supports env var expansion
    secret_key: "${S3_SECRET_KEY}"
    ttl_override: "24h"              # Optional: keep L2 entries longer than global TTL

  tiered:                            # Promotion/demotion behavior
    promote_on_hit: true             # Promote L2 hits back to L1 (default: true)
    l1_promotion_ttl: "5m"          # TTL for promoted entries (default: "5m")

Metrics: The s3-cache build adds Prometheus counters for L1/L2 hit rates, promotion/demotion counts, and an S3 operation latency histogram (continuum_cache_s3_latency_seconds).

Using Redis as L1: Combine the redis-cache and s3-cache features to use Redis as the L1 tier:

cargo build --release --features redis-cache,s3-cache

response_cache:
  backend: tiered
  l1:
    type: redis                      # Use Redis as L1 for cross-instance sharing
    max_value_size: 1048576
  redis:                             # Shared Redis config reused by l1
    url: "redis://localhost:6379"
    pool_size: 8
  l2:
    type: s3
    endpoint: "https://s3.example.com:8080"
    bucket: "llm-response-cache"
    access_key: "${S3_ACCESS_KEY}"
    secret_key: "${S3_SECRET_KEY}"

Limitations¶

• Streaming cache is supported for OpenAI-compatible backends only. Anthropic and Gemini native streaming formats are not cached.
• The Anthropic (/anthropic/v1/messages) and OpenAI (/v1/chat/completions) endpoints each maintain the same shared cache instance when response_cache.enabled: true.
• Without the Redis backend configured, cache is stored in-memory and does not survive server restarts.

KV Cache Index Section¶

Enables real-time KV cache state tracking for precise overlap-scored routing. When a vLLM backend processes a request, it emits SSE events describing which token prefixes are cached. The router subscribes to these events, maintains an index of cache state across all backends, and uses this data to route requests to the backend most likely to have relevant KV cache data.

kv_cache_index:
  enabled: true                      # Enable KV cache index (default: false)
  backend: memory                    # Index backend: "memory" (default) or "redis"
  max_entries: 100000                # Maximum prefix hash entries tracked
  entry_ttl_seconds: 600             # TTL for index entries in seconds
  scoring:                           # Scoring weights for backend selection
    overlap_weight: 0.6              # Weight for cache overlap signal (0.0-1.0)
    load_weight: 0.3                 # Weight for backend load signal (0.0-1.0)
    health_weight: 0.1               # Weight for backend health signal (0.0-1.0)
    min_overlap_threshold: 0.3       # Minimum overlap to activate KV-aware routing
    gpu_tier_weight: 1.0             # Tier multiplier for GPU-resident (hot) cached data
    storage_tier_weight: 0.6         # Tier multiplier for storage-offloaded (warm) cached data
  storage_offloading:                # Tiered storage awareness (GPU hot / storage warm)
    enabled: false                   # Enable storage tier tracking (default: false)
    treat_eviction_as_offload: true  # Treat eviction as offload to warm storage (default: true)
  event_sources:                     # vLLM backend KV event stream endpoints
    - backend_name: "vllm-1"
      endpoint: "http://vllm-1:8000/v1/kv_events"
      reconnect_interval_ms: 5000
    - backend_name: "vllm-2"
      endpoint: "http://vllm-2:8000/v1/kv_events"
      reconnect_interval_ms: 5000

How It Works¶

1. Event consumption: The router subscribes to SSE event streams from configured vLLM backends. Each event reports a CacheCreated, CacheEvicted, CacheOffloaded, CacheReloaded, or CachePurged action with a prefix hash and token count.
2. Index update: Events update an in-memory (or Redis-backed) index mapping prefix hashes to backends with their cached token counts and storage tier (GpuHot or StorageWarm).
3. Scoring: On each request, the overlap scorer queries the index for the request's prefix key and computes a weighted score combining cache overlap (adjusted by tier multiplier), backend load, and backend health.
4. Routing decision: If the best score exceeds min_overlap_threshold, the scorer selects that backend. Otherwise, the configured selection strategy (e.g., PrefixAwareHash) is used as fallback.

Scoring Formula¶

final_score = overlap_weight * (raw_overlap * tier_multiplier)
            + load_weight * (1 - load_ratio)
            + health_weight * health_score

Where:

• raw_overlap = backend_token_count / max_token_count_across_backends (0.0 to 1.0)
• tier_multiplier = gpu_tier_weight for GPU-resident data, or storage_tier_weight for offloaded data
• load_ratio = in_flight_requests / max_in_flight (0.0 to 1.0)
• health_score = backend success_rate (0.0 to 1.0)

When storage_offloading.enabled is false, all data is treated as GpuHot and tier_multiplier always equals gpu_tier_weight (default: 1.0).

Configuration Options¶

Redis Backend¶

When backend: redis, the KV index uses the shared Redis connection from response_cache.redis. This enables multiple router instances to share the same index for consistent routing decisions across the fleet.

response_cache:
  redis:
    url: "redis://localhost:6379"
    pool_size: 8

kv_cache_index:
  enabled: true
  backend: redis    # Uses the Redis config from response_cache.redis

Admin Endpoints¶

• GET /admin/kv-index/stats — Index size, event rates, connection status, scoring distribution
• GET /admin/kv-index/backends — Per-backend cache state summary
• POST /admin/kv-index/clear — Clear the index (for debugging)

See Admin API for full response schemas.

Limitations¶

• Event sources only support vLLM-compatible SSE endpoints currently.
• Changing backend between memory and redis requires a restart.
• The KV index is an optional enhancement; its unavailability does not break routing.

Logging Section¶

Configures logging output:

logging:
  level: "info"                   # trace, debug, info, warn, error
  format: "json"                  # json, pretty

Log Levels:

• trace: Extremely verbose, includes all details
• debug: Detailed debugging information
• info: General operational information
• warn: Warning messages and potential issues
• error: Error conditions only

Log Formats:

• json: Structured JSON logging (recommended for production)
• pretty: Human-readable format (good for development)