KV Cache Optimization¶

Continuum Router provides a four-tier KV cache optimization system that reduces redundant computation on LLM backends. Each tier is an independent configuration toggle rather than a stacked layer. Enabled tiers run in a fixed order: Tier 2 (response cache), Tier 4 (KV cache index scorer), and finally the configured selection_strategy, which acts as Tier 1 when set to PrefixAwareHash.

Table of Contents¶

• Overview
• Four-Tier Caching Strategy
• Tier 1: Prefix-Aware Sticky Routing
• Tier 2: Response Cache
• Tier 3: Shared External Cache
• Tier 4: Backend KV Cache Index
• Backend Selection Pipeline
• Gemini Context Cache
• Configuration Reference
• Metrics
• Admin Endpoints
• Deployment Guide
• Performance Characteristics

Overview¶

Modern LLM inference engines (vLLM, TensorRT-LLM, SGLang) maintain a KV cache in GPU memory that stores attention key-value tensors computed for the token prefix of a request. When the same prefix appears again, the engine skips recomputing those tensors, saving significant GPU compute time for long system prompts or repeated context.

Continuum Router maximizes KV cache reuse across backends through four complementary mechanisms:

1. Prefix-Aware Sticky Routing: routes requests sharing the same prompt prefix to the same backend via consistent hashing, keeping GPU KV cache warm.
2. Response Cache: serves repeated deterministic requests directly from router memory or Redis without hitting any backend at all.
3. Shared External Cache: stores response cache state in Redis/Valkey so multiple router instances share the same cache entries.
4. Backend KV Cache Index: tracks which backends actually hold GPU-resident KV tensors for recent prefixes, enabling fine-grained routing decisions informed by real cache state.

Four-Tier Caching Strategy¶

flowchart TD
    Client([Client Request])
    RC{Response\nCache hit?}
    PRL{Prefix key\navailable?}
    REG{KV index scorer\nregistered?}
    SCORE[Composite Score\noverlap + load + health]
    THRESH{Best combined\nscore > 0.3?}
    STRATEGY[Configured selection_strategy]
    PAH{PrefixAwareHash:\nprefix key present?}
    CHWBL[CHWBL Hash Ring\nPrefix → Backend, load cap]
    MHASH[Model-name consistent hash\nno load cap]
    BACKEND([Selected Backend])
    STORE_RC[Store Response\nin Cache]

    Client --> RC
    RC -->|HIT| Client
    RC -->|MISS| PRL
    PRL -->|Yes| REG
    REG -->|Yes| SCORE
    SCORE --> THRESH
    THRESH -->|Yes| BACKEND
    THRESH -->|No| STRATEGY
    REG -->|No| STRATEGY
    PRL -->|No| STRATEGY
    STRATEGY -->|PrefixAwareHash| PAH
    STRATEGY -->|any other strategy| BACKEND
    PAH -->|Yes| CHWBL
    PAH -->|No| MHASH
    CHWBL --> BACKEND
    MHASH --> BACKEND
    BACKEND --> STORE_RC
    STORE_RC --> Client

The four tiers operate independently:

• Tier 2 intercepts eligible requests before any backend is contacted, using Tier 3 (Redis/Valkey) when configured as its shared storage backend.
• Tier 4 runs first during backend selection whenever both prefix_routing.enabled (which computes the prefix key) and kv_cache_index.enabled (which maintains the index) are true, regardless of selection_strategy.
• Tier 1 runs when the Tier 4 scorer does not select a backend and selection_strategy is set to PrefixAwareHash.

Tier 1: Prefix-Aware Sticky Routing¶

Tier 1 routes requests that share a common prompt prefix to the same backend, maximizing the probability that the GPU KV cache on that backend is already warm for those tokens.

Prefix Key Extraction¶

For each incoming chat completion request, the router extracts a prefix key: a 32-byte SHA256 digest that uniquely identifies the semantic anchor of the request.

The extraction logic handles both OpenAI and Anthropic request formats:

The hash is computed as:

# With system prompt:
SHA256(model_bytes ++ "\x00" ++ "S" ++ system_bytes[:max_prefix_length])

# Without system prompt (first message fallback):
SHA256(model_bytes ++ "\x00" ++ "M" ++ first_msg_bytes[:max_prefix_length])

The \x00 separator prevents length-extension collisions between the model name and content. The tag bytes S/M prevent identical text from hashing to the same value when it appears as a system prompt versus a first user message.

The max_prefix_length parameter (default: 1024 bytes) truncates the content before hashing, with UTF-8 boundary awareness to avoid splitting multibyte characters.

Implementation: src/core/prefix_key.rs, src/core/hashing.rs

Consistent Hashing with Bounded Loads (CHWBL)¶

The PrefixAwareHash selection strategy uses a consistent hash ring to map prefix keys to backends. Simple consistent hashing can produce uneven load distribution when some prefixes are far more popular than others. Continuum Router addresses this with the Consistent Hashing with Bounded Loads (CHWBL) algorithm.

CHWBL adds a load cap: a backend can handle at most (1 + epsilon) * average_load requests simultaneously. When a backend is at its load cap, the request overflows to the next node clockwise on the ring.

epsilon = 0.25  →  a backend can be at most 25% above average load

The ring is populated with virtual_nodes (default: 150) virtual replicas per backend to improve key distribution uniformity. virtual_nodes is applied to the live pool on hot-reload: changing it rebuilds the consistent-hash ring at the new size without a restart.

When no prefix key exists¶

The CHWBL ring is used only when the request carries a prefix key. Two cases produce no key:

• prefix_routing.enabled is false (the default). Both request_prefix_key and typed_request_prefix_key return None immediately, so no request produces a prefix key.
• prefix_routing.enabled is true, but the request body contains neither a system prompt nor any messages to hash.

In both cases, PrefixAwareHash hashes the model name instead and walks the ring using plain consistent hashing without a bounded-load cap. It selects the first node clockwise from the model hash and never overflows to a less-loaded backend, routing all requests for a given model to the same backend while pool membership remains unchanged.

Because PrefixAwareHash requires prefix_routing.enabled: true to perform prefix-aware routing, continuum-router config validate emits a warning at selection_strategy when PrefixAwareHash is paired with prefix_routing.enabled: false. The router also logs this warning at startup and on hot reload.

Routing decision labels¶

continuum_prefix_routing_requests_total{strategy=...} is recorded once per selection, and only when the PrefixAwareHash arm actually ran. Other strategies record nothing on this metric family.

• prefix_hash: the prefix-hashed owner took the request. This covers the ordinary case (the owner was under the load cap) and the full-lap case where every backend was at the cap and the ring walk kept the owner to preserve prefix locality.
• overflow: the owner was at or over the load cap, so a later node clockwise on the ring took the request.
• fallback: the request carried no prefix key, so the model name was hashed and the ring was walked with no load cap. See When no prefix key exists above.

Anthropic Cache Control Injection¶

When anthropic_cache_control_injection: true is set, the router automatically adds cache_control: { type: "ephemeral" } markers to the system prompt of native-Anthropic requests. This activates Anthropic's server-side prompt caching, which is distinct from the router-level KV cache but complementary to it.

This flag is a fleet-wide master enable and is read per-request (hot-reloadable). Its precedence with the per-backend anthropic_auto_cache_control:

• Per-backend anthropic_auto_cache_control governs by default and is true when unset, so native-Anthropic backends inject automatically out of the box.
• prefix_routing.anthropic_cache_control_injection: true forces injection on even for a backend that set anthropic_auto_cache_control: false.
• Its default (false) is a strict no-op: the per-backend setting alone decides.
• Injection only ever touches native-Anthropic requests and never other providers. It targets the last system block only, and if the request already carries any cache_control (system, message, or tool), injection is skipped to preserve the caller's explicit caching intent.

Tier 2: Response Cache¶

The response cache stores complete LLM responses for deterministic requests, allowing the router to serve repeated queries without contacting any backend.

Cache Eligibility¶

A request is eligible for caching when all of the following are true:

• temperature is 0 or absent
• The request does not use streaming (or uses streaming with buffering enabled)
• The accumulated response size is within max_response_size

Requests with non-zero temperature are probabilistic and are never cached. The response header X-Cache: HIT, X-Cache: MISS, or X-Cache: BYPASS indicates the cache disposition.

Cache Key Computation¶

The cache key is a SHA256 hash over all parameters that affect LLM output:

SHA256(
    model,
    "\x00",
    SHA256(messages),   // pre-hashed messages array
    "\x00",
    temperature_bytes,
    "\x00",
    SOME/NONE_tag + max_tokens_bytes,
    "\x00",
    SOME/NONE_tag + top_p_bytes,
    "\x00",
    SOME/NONE_tag + tenant_id_bytes,
)

SOME/NONE tag bytes prevent collisions between None and Some(0) for optional parameters. The tenant ID is included to provide multi-tenant isolation: tenants cannot read each other's cached responses.

Implementation: src/infrastructure/cache/response_cache.rs

Streaming Cache¶

For streaming responses, the router accumulates the SSE stream into a buffer up to max_stream_buffer_size (default: 10 MiB). If the complete stream fits within the limit, the buffer is stored as a single serialized blob and replayed as a synthetic SSE stream on cache hits.

Cache Eviction¶

The in-memory backend uses LRU eviction when the entry count reaches capacity. The Redis backend relies on TTL-based expiration managed by Redis itself.

Semantic Cache¶

In addition to exact and prefix lookups, an optional semantic cache can serve a deterministic request from an earlier response whose final user message is semantically similar. It is consulted only after exact and prefix lookups miss, and only when all of the following conditions hold:

• response_cache.semantic.enabled: true, with backend and embedding_model configured
• control_plane.optimization.semantic_cache_enabled: true on the router
• Both semantic_cache and exact_cache enabled in the Hub optimization policy for the organization, on a tier with cache_enabled
• A non-streaming /v1/chat/completions request with temperature: 0 whose final message is a user turn containing at most max_input_bytes of text

Embedding lookup and backend visibility¶

Semantic lookup runs after the guardrail input gate, so blocked prompts are never sent to the embedder and transform verdicts are applied before embedding. Each eligible request makes a single POST /v1/embeddings call to the configured backend, bounded by embedding_timeout_ms.

On a miss, the router retains the computed vector and indexes the backend response under it once the response arrives. If the embedding call fails or times out, the router treats it as a cache miss without failing the chat request.

The pinned embedding backend follows the same visibility rules as the LLM classifier and guardrail transport: an internal: true backend is reachable when explicitly named, whereas an enabled: false backend is rejected.

Partitioning and isolation¶

Cache isolation is enforced through partitioning. Cosine similarity is computed only among entries sharing the same partition digest, which hashes together:

• The caller's cache identity (route-scoped local identity plus Hub key ID)
• The target model and the embedding model
• All response-shaping request fields (tools, response format, sampling bounds, and seed)
• The entire conversation history preceding the final user turn, including the system prompt

Because all of these inputs are folded into the partition digest, different API keys, system prompts, or tool definitions never share semantic cache entries.

Serving, storage, and TTL¶

A candidate entry is served when its cosine similarity meets threshold (or the Hub policy's semantic_similarity_bps when threshold is omitted). Semantic hits return X-Cache: HIT and X-Cache-Mode: semantic, and record cache_hit_type = semantic in the Hub usage record without charging token rate limits or budgets.

Cached response bodies are stored in the response cache under the sem: keyspace, while the vector index is maintained in memory per router (max_entries, with oldest-first eviction). Entry TTL follows the Hub policy's cache_ttl_secs when present, falling back to semantic.ttl and then response_cache.ttl. Cross-router vector indexes, external vector databases, and streaming semantic cache hits are not supported.

Because semantic response bodies share response_cache.capacity with exact and prefix entries, size capacity for the combined volume or keep semantic.max_entries small enough to avoid excessive eviction pressure.

Implementation: src/infrastructure/cache/semantic/ (partition, index, embedding shapes) and src/proxy/semantic_cache.rs (serving path).

Tier 3: Shared External Cache¶

The shared external cache provides a CacheStore trait abstraction over Redis/Valkey, allowing multiple router instances to share response cache state.

CacheStore Trait¶

pub trait CacheStore: Send + Sync + 'static {
    async fn get(&self, key: &str) -> CacheStoreResult<Option<Vec<u8>>>;
    async fn set(&self, key: &str, value: &[u8], ttl: Duration) -> CacheStoreResult<()>;
    async fn delete(&self, key: &str) -> CacheStoreResult<()>;
    async fn clear(&self) -> CacheStoreResult<()>;
    async fn stats(&self) -> CacheStoreStats;
}

Implementations: InMemoryCacheStore (default, LRU + TTL), RedisCacheStore (Redis/Valkey with connection pooling).

Implementation: src/infrastructure/cache/store.rs

Redis Backend¶

RedisCacheStore uses deadpool-redis for connection pooling. All keys are namespaced with a configurable prefix (default: cr:resp:) to avoid collisions with other applications sharing the same Redis instance.

Key namespacing format:

cr:resp:<sha256-cache-key-hex>

Operations use SET EX for writes and GET for reads, with configurable command timeouts (default: 1s).

Automatic Fallback¶

When Redis is unreachable, RedisCacheStore transparently activates an in-memory fallback cache. The fallback is activated on the first connection failure and a background health-monitor task (running every 30 seconds) attempts to restore the Redis connection. On recovery, the flag is cleared and subsequent operations go back to Redis.

The continuum_cache_fallback_active metric (value 1) indicates that fallback mode is currently active.

Connection Pool Sharing¶

The deadpool_redis::Pool is stored as an Arc in AppState and shared between the response cache and the KV cache index (Tier 4). This avoids double-counting connections and simplifies configuration: both consumers reuse the same pool credentials.

Tier 4: Backend KV Cache Index¶

Tier 4 tracks in real time which backends hold GPU-resident KV tensors for specific token prefix hashes. This enables routing decisions based on actual GPU cache state rather than statistical affinity.

Event Consumption¶

The continuum-kv-listener binary acts as the event producer bridge. It ingests vLLM and SGLang msgpack records over ZMQ (supporting both array-encoded sequences from vLLM <= 0.28 / SGLang and map-encoded tagged-struct objects from vLLM 0.29+) or polls native trtllm-serve JSON events over HTTP.

The listener resolves block chains to the router's prefix hash and exposes an SSE stream per backend (for example, http://kv-listener.internal:7817/events/vllm-1). Inside the router, KvEventConsumerManager spawns a background Tokio task per backend to subscribe to the SSE stream and update the KV index.

Block chain tracking and memory bounds¶

Instead of copying the full token chain into every block, the listener stores only each block's own token segment and parent reference:

• max_chain_tokens (default: 16384) caps the token ancestry reconstructed for /detokenize. A block retains its token segment only while its chain fits within max_chain_tokens; longer chains stay sequenced under the metadata ceiling as unattributable.
• max_tracked_blocks (default: 100000) bounds block metadata, and max_retained_tokens (default: 16777216, or 64 MiB of 4-byte IDs) bounds retained token IDs, both using oldest-entry eviction.
• Reaching either cap purges only prefixes whose last tracked reference was retired, leaving unrelated prefixes on the backend intact. Total retained token IDs per source never exceed min(max_retained_tokens, max_tracked_blocks * max_chain_tokens) (64 MiB by default).
• Salt-echo and native TensorRT-LLM sources verify prefix identity directly from the salt without retaining or reconstructing tokens.

Engine block_size semantics¶

Although a standard single-block event carries block_size tokens, block_size is engine-authored and unbounded, so the listener validates token counts per engine:

• SGLang (python/sglang/srt/mem_cache/events.py at 8055ccd2cd): emits one event per page with block_size equal to the page length and coalesces only events of equal block_size, so len(token_ids) == block_size * len(block_hashes).
• vLLM (vllm/v1/core/block_pool.py at 713ec07655): sends one KV-block span per hash on its full-block path, but omits null or masked blocks from block_hashes while keeping the token span contiguous, so sliding-window or hybrid attention hashes can carry a multi-block span. Its partial-block path sends a single hash_block_size segment.
• Mismatch handling: if a single-block event has a non-empty token_ids length that differs from a non-zero block_size, the listener tracks the block without tokens (leaving its descendants unattributable), increments a counter per event, and logs a warning once per source.

Native TensorRT-LLM events¶

For TensorRT-LLM sources, a created event resets the listener's tracked block state, stored events create or extend block chains, and removed events evict their resolved prefixes. The HTTP poller uses the event ID to reject duplicates and clear stale state after sequence gaps. Dynamo deployments that already publish vLLM-compatible ZMQ events should continue using the ZMQ source rather than adding a duplicate HTTP poller.

Event types:

Each event carries a prefix_hash (hex string) and an optional token_count indicating how many tokens are cached for that prefix on that backend.

SSE parsing details:

• The consumer preferentially uses the SSE event: field to determine event type; the JSON event field is a fallback.
• Both the line buffer and the accumulated data: payload of a single record are capped at 1 MiB (MAX_SSE_BUFFER_SIZE). If a peer sends data: lines without a terminating blank line and exceeds 1 MiB, the consumer drops the stream, reconnects with backoff, and resumes from the last forwarded event ID.
• On connection failure or stream end, the consumer applies exponential backoff (initial: 1s, max: 60s) before reconnecting.

Implementation: src/infrastructure/kv_index/event_consumer.rs

Producer bridge implementation: crates/continuum-kv-listener/

Push Mode: Listener Dials the Router¶

By default, the router pulls each backend's events from the listener's SSE endpoint, which requires network reachability from the router to the listener. When the listener runs inside a private GPU cluster and the router runs outside it, set transport: event_bus on the source and configure an event_bus section on the listener.

In push mode, the listener initiates an outbound HTTP/2 + TLS connection to the router's event bus and pushes per-backend event streams. The listener's local SSE server remains active, and any source without transport: event_bus continues to use SSE pull. Push mode requires a router binary built with the dynamo-event-bus feature (included in official release binaries).

sequenceDiagram
    participant L as continuum-kv-listener
    participant R as Router event bus
    participant C as kv_cache_index consumer (vllm-1)
    L->>R: POST /bus/v1/session, hello (role kv_listener, sources vllm-1, vllm-2)
    R-->>L: welcome
    C->>R: open_announced_stream(vllm-1, last_event_id)
    R->>L: subscribe (announced_source vllm-1, last_event_id)
    L->>R: POST /bus/v1/streams/{id}, exact SSE bytes
    R->>C: SSE bytes, parsed and tagged vllm-1

The router and listener configurations are paired by client_id and backend_name. Configure a role: kv_listener entry in dynamo.event_bus.brokers[] on the router and set transport: event_bus on each matching event source:

# Router
dynamo:
  event_bus:
    enabled: true
    listen: "0.0.0.0:9443"
    tls:
      cert: "/etc/continuum-router/event-bus/tls.crt"
      key: "/etc/continuum-router/event-bus/tls.key"
    brokers:
      - id: "kv-listener-a"                 # must equal the listener's event_bus.client_id
        role: kv_listener
        token_hash: "sha256:<64 lowercase hex>"   # printf %s "$TOKEN" | sha256sum
        backends: ["vllm-1", "vllm-2"]      # the only backends this credential may announce

kv_cache_index:
  enabled: true
  event_sources:
    - backend_name: vllm-1
      transport: event_bus                  # no endpoint
      reconnect_interval_ms: 5000
    - backend_name: vllm-2
      transport: event_bus

# continuum-kv-listener (config.kv-listener.yaml)
listen: "127.0.0.1:7817"
event_bus:
  url: "https://router.example.com:9443"    # https only; host[:port], default port 443
  ca_file: "/etc/continuum/bus-ca.pem"      # omit for public web roots
  client_id: "kv-listener-a"
  credential_file: "/etc/continuum/bus-token"   # or credential_env; never inline
sources:
  - backend_name: "vllm-1"                  # must be a valid source id: 1-64 of [A-Za-z0-9_-]
    engine: "vllm"
    zmq_endpoint: "tcp://127.0.0.1:5557"
    zmq_mode: "bind"
    detokenize_endpoint: "http://vllm-1:8000/detokenize"
    model: "meta-llama/Llama-3.1-8B-Instruct"
    allowed_publishers: ["127.0.0.1"]
  # ... vllm-2 likewise

See config.kv-listener.yaml.example and continuum-kv-listener(1) for the full set of listener options (tls_server_name, mTLS client_cert_file and client_key_file, credential_env, reconnect_backoff, drain_deadline, and connect_timeout).

Stream lifecycle and reconnection¶

• Resume from last event ID: The consumer for backend X opens only source X, parses the stream with the standard SSE parser, tags events with X, and passes the last processed event ID on reconnect so buffered events are neither replayed nor skipped.
• Backoff and session wait: On stream errors, the consumer backs off from reconnect_interval_ms up to max(reconnect_interval_ms, 60s). When no active listener session has announced the source (no_session), retries stay fixed at reconnect_interval_ms so the stream opens promptly once the listener connects.
• Backpressure and draining: If a consumer falls behind the live feed, the listener closes the stream with buffer_overflow rather than dropping events silently, and the router reconnects using its last event ID. On SIGTERM, the listener sends draining so the router immediately switches to another active session. Hot-reloading kv_cache_index restarts bus sources alongside SSE sources.

Wire parity¶

Push streams emit the exact same SSE byte format as GET /events/{backend} and use the same router-side SSE parser. Both the listener test suite and the router parity test verify output against the shared golden fixture at crates/continuum-kv-listener/tests/fixtures/push_stream_golden.sse.

Security and authorization¶

• TLS and scoped credentials: Unlike the plain HTTP SSE port, the event bus requires TLS and authenticates every session via bearer token hash (SHA-256) or mTLS client certificate (DNS SAN matching the entry id).
• Role and backend enforcement: A broker credential cannot open a kv_listener session (and vice versa). Each listener may announce only the backends listed in its brokers[] entry, and hot-reloading a narrower allow-list terminates active sessions with forbidden.
• Hub config sync exclusion: Event bus sources are local to the router and are not exported to or managed by Control Plane Hub config sync.

Index Structure¶

Events are fed into a KvCacheIndex implementation, which maintains a mapping:

prefix_hash → {backend_id → token_count}

The token count acts as the score: a backend holding 2048 cached tokens for a prefix ranks higher than one holding 512.

Two implementations are provided:

InMemoryKvIndex¶

• DashMap<String, PrefixEntry> for lock-free concurrent reads
• LRU eviction when entry count reaches max_entries (evicts the oldest 10% of entries)
• TTL-based expiration checked lazily on query_backends()
• Periodic cleanup_expired() removes stale entries proactively
• Default: 100,000 max entries, 300s TTL

RedisKvIndex¶

• Stores each prefix as a Redis sorted set: ZADD cr:kvidx:<prefix_hash> <token_count> <backend_id>
• EXPIRE is pipelined with ZADD in a single atomic round-trip
• ZREVRANGEBYSCORE +inf -inf WITHSCORES returns backends ranked by descending score
• Enables sharing of KV index state across multiple router instances
• Key prefix: cr:kvidx:

Implementation: src/infrastructure/kv_index/index.rs

Storage Tier Awareness¶

When storage_offloading.enabled is true, the index tracks two storage tiers for each (prefix, backend) entry:

Tier transitions from events:

The treat_eviction_as_offload option controls whether generic cache_evicted events (which do not carry tier information) are treated as offloads to warm storage or as permanent removals. This is useful when vLLM backends emit only cache_created and cache_evicted events without the explicit cache_offloaded event type.

Implementation: src/infrastructure/kv_index/types.rs

Overlap Scoring¶

The KvOverlapScorer implements the BackendScorer trait and computes a composite score for each backend that holds cached data for the request's prefix:

final_score = overlap_weight   * (raw_overlap * tier_multiplier)
            + load_weight      * (1.0 - load_ratio)
            + health_weight    * health_score

Where:

• raw_overlap = backend_token_count / max_token_count_across_backends (0.0 to 1.0)
• tier_multiplier = gpu_tier_weight for GpuHot data, or storage_tier_weight for StorageWarm data
• load_ratio = backend_in_flight / max_in_flight_across_backends (0.0 to 1.0)
• health_score = backend_success_rate (0.0 to 1.0)

Any backend with no index entry for the requested prefix receives a score of 0.0 and is excluded from ranking. Load and health signals only break ties among backends that actually hold cached prefix data; otherwise, an idle healthy backend with no cached tokens would score load_weight + health_weight (0.4 by default), exceed the 0.3 selection threshold, and override the configured routing strategy.

Where the weights come from¶

The six scoring parameters are loaded from kv_cache_index.scoring when the scorer is registered at startup and logged once at INFO. Because kv_cache_index requires a restart on configuration changes, editing scoring and hot-reloading does not rebuild the scorer.

Default weights:

The three main weights (overlap_weight, load_weight, health_weight) must sum to 1.0 (validated at startup).

The tier multipliers (gpu_tier_weight, storage_tier_weight) scale raw_overlap before the weighted sum is computed, so a StorageWarm backend scores lower than a GpuHot backend holding the same number of cached tokens.

Load and health inputs¶

The load signal reads the backend pool's live in-flight request tracker (the same gauge used by CHWBL). During prepare(), the scorer captures a single snapshot of the in-flight tracker alongside the KV index query and caches both for 100 ms, ensuring all candidate backends in a scoring pass are compared against a consistent load view.

The health signal reads the backend's success rate synchronously from the shared statistics map, substituting a neutral 1.0 if the lock is contended.

Minimum overlap threshold¶

min_overlap_threshold (default: 0.3) filters each candidate backend based on its tier-weighted normalized overlap (raw_overlap * tier_multiplier). Any backend below this threshold scores 0.0.

Because raw_overlap is normalized against the backend with the highest cached token count:

• The top GpuHot holder has raw_overlap = 1.0 and a tier-weighted overlap of gpu_tier_weight, passing the gate whenever gpu_tier_weight >= min_overlap_threshold (1.0 >= 0.3 by default).
• A backend holding only a StorageWarm copy passes only when storage_tier_weight >= min_overlap_threshold (0.6 >= 0.3 by default). Setting storage_tier_weight below min_overlap_threshold disables routing to StorageWarm-only holders.

If gpu_tier_weight is configured below min_overlap_threshold, no backend can ever pass the threshold and routing always falls back to selection_strategy. When both prefix_routing.enabled and kv_cache_index.enabled are true, continuum-router config validate and router startup emit a warning on kv_cache_index.scoring.gpu_tier_weight.

When no candidate passes min_overlap_threshold, all backends score 0.0. Because the pool requires a combined score strictly greater than 0.3 (DEFAULT_SCORER_THRESHOLD), the router falls back to the configured selection_strategy.

Async preparation and tier reporting¶

Because KvCacheIndex::query_backends() is asynchronous while BackendScorer::score() is synchronous, the scorer uses a two-phase design: prepare() queries the index and captures the in-flight snapshot (cached for 100 ms), and score() reads synchronously from that cached pass.

Implementation: src/infrastructure/kv_index/scorer.rs

To report which cache tier won the selection, BackendScorer::score_with_tier returns both the score and the holder tier (gpu_hot, storage_warm, or none) in a single read from the request's validated cache pass. The pool attaches the winning backend's tier to the selection result and records it as the tier label on continuum_kv_index_routing_decisions_total, avoiding any race with concurrent requests updating the scorer cache.

Backend Selection Pipeline¶

When a chat completion request arrives, the router executes the following pipeline:

sequenceDiagram
    participant C as Client
    participant R as Router
    participant RC as Response Cache
    participant PK as Prefix Key
    participant KVI as KV Index
    participant CHWBL as CHWBL Ring
    participant B as Backend

    C->>R: POST /v1/chat/completions
    R->>RC: Lookup cache key (T=0 only)
    alt Cache HIT
        RC-->>R: Cached response
        R-->>C: 200 OK (X-Cache: HIT)
    else Cache MISS
        R->>PK: Extract prefix key
        alt Prefix key available and KV index scorer registered
            R->>KVI: prepare(prefix_hash)
            KVI-->>R: Backend scores from index
            R->>R: Score = overlap + load + health (holders only)
            alt Best combined score > 0.3
                R->>B: Forward request (KV-aware routing)
            else At or below threshold
                R->>R: Run configured selection_strategy
                R->>CHWBL: PrefixAwareHash: map prefix → backend (load cap)
                R->>B: Forward request
            end
        else Prefix key available, no scorer registered
            R->>R: Run configured selection_strategy
            R->>CHWBL: PrefixAwareHash: map prefix → backend (load cap)
            R->>B: Forward request
        else No prefix key
            R->>R: Run configured selection_strategy
            R->>CHWBL: PrefixAwareHash without a prefix key: hash the model name (no load cap)
            R->>B: Forward request
        end
        B-->>R: Response
        R->>RC: Store response (T=0)
        R-->>C: 200 OK (X-Cache: MISS)
    end

The KV overlap scorer runs before selection_strategy and complements prefix-aware routing:

• When at least one eligible backend holds cached tokens for the prefix and its combined score exceeds 0.3, the KV overlap scorer selects the backend.
• Otherwise, the router falls back to the configured selection_strategy (PrefixAwareHash uses the CHWBL ring, while RoundRobin, WeightedRoundRobin, LeastLatency, Random, and ConsistentHash run their respective algorithms).

Because Tier 4 depends only on prefix_routing.enabled and kv_cache_index.enabled, pairing RoundRobin with both flags enabled provides full KV-aware routing with round-robin as the fallback when no backend holds the prefix.

Gemini Context Cache¶

Tiers 1 through 4 target backends with a GPU-resident KV cache (vLLM, TensorRT-LLM, SGLang). Because Google's Generative Language API does not expose a KV event stream, type: gemini backends use a separate opt-in mechanism where the router manages Google's server-side cachedContents resources automatically.

Overview¶

When enabled, the router caches large, recurring system-prompt prefixes sent to type: gemini backends as cachedContents resources and reuses them across requests. OpenAI-compatible clients continue sending standard requests, and the router rewrites them before forwarding:

• OpenAI-compatible endpoint: references the cache via extra_body.google.cached_content = "cachedContents/...".
• Native generateContent endpoint: sets the top-level cachedContent field and omits systemInstruction (since the system prompt is stored in the cached resource).

Google bills cached tokens at a discounted input rate plus an hourly storage fee per token, and enforces a per-model minimum token count (roughly 2,048 tokens for Gemini 2.5 models and 4,096 tokens for Gemini 3.x models). Because context caching is cost-effective only when prefixes are both large and frequently reused, gemini_context_cache.enabled defaults to false with a short ttl.

The router approximates Google's token minimum using a byte threshold (min_prefix_bytes) without tokenizing the prompt. If Google rejects a prefix as too short, the router negative-caches that prefix for negative_ttl_seconds to avoid repeated failed creation calls. When gemini_context_cache.enabled is false, no maps or background tasks are allocated.

Cached token metering¶

Cache hits are metered at the cached input rate whether they come from a router-managed cachedContents resource or from Gemini 2.5+ implicit caching (which populates usageMetadata.cachedContentTokenCount automatically). Across API surfaces, token counts are reported as follows:

• /v1/chat/completions: prompt_tokens remains inclusive of cached tokens, and usage.prompt_tokens_details.cached_tokens reports the cached portion (both in buffered responses and in the final usage chunk of streaming responses).
• /v1/responses: usage.input_tokens_details.cached_tokens reports the cached token count.
• /anthropic/v1/messages: follows Anthropic's convention by reporting non-cached input_tokens alongside cache_read_input_tokens.

Usage records and local rate/budget enforcement charge input_tokens = prompt_tokens - cache_read against input_tpm and monthly budgets, recording the cached portion in cached_input_tokens with cache_hit: true.

Request Lifecycle¶

A request is eligible when it targets a type: gemini backend and carries a system prompt of at least min_prefix_bytes:

1. Prefix digest: Computes a SHA-256 digest over the model name and the full system prompt (without truncation), matching the structure in src/core/prefix_key.rs.
2. Map hit: If an active entry exists, the router injects the cachedContents reference and removes the duplicate system prompt. When extend_on_hit: true and remaining TTL is below 20%, a background task sends a PATCH request to extend the TTL.
3. Map miss: If the prefix is not cached (and not negative-cached), the router forwards the request unmodified and spawns a single-flight background task (POST /v1beta/cachedContents with model, systemInstruction, and ttl), so the first request pays no creation latency.
4. Creation rejected: If Google rejects creation with HTTP 400 (for example, below the model's minimum token count), the router negative-caches the prefix for negative_ttl_seconds.
5. Eviction: Entries exceeding max_entries are evicted via LRU. Both LRU eviction and the Admin clear endpoint send a best-effort DELETE to Google to stop storage billing early. The negative cache shares the same max_entries limit, sweeping expired entries before dropping new rejections.

Configuration Reference¶

gemini_context_cache:
  enabled: false

  # Minimum system-prompt size (UTF-8 bytes) to attempt caching (default: 16384)
  # Must be >= 1024. The router does not tokenize; this is a byte-based proxy
  # for Google's per-model minimum cacheable token count.
  min_prefix_bytes: 16384

  # TTL applied when creating a cachedContents resource (default: "10m")
  # Must parse as a duration and be >= 60s
  ttl: "10m"

  # Extend the TTL in the background when a hit lands within 20% of expiry (default: false)
  extend_on_hit: false

  # Maximum tracked entries before LRU eviction (default: 1000)
  # Range: 10 to 1,000,000
  max_entries: 1000

  # Timeout for the cachedContents creation call, in milliseconds (default: 5000)
  # Range: 500 to 60000
  create_timeout_ms: 5000

  # TTL for negative-caching a prefix Google rejected, in seconds (default: 600)
  # Must be >= 1
  negative_ttl_seconds: 600

  # Prefer the Gemini Native API (generateContent / streamGenerateContent) over
  # the OpenAI-compatible /v1beta/openai/chat/completions endpoint for all
  # Chat Completions requests routed to type: gemini backends (default: false)
  prefer_native_api: false

Scope: only chat completions on type: gemini backends are covered. The /v1/responses Gemini converter, image generation, and embeddings paths do not participate. Proxies or self-hosted endpoints that mimic the Generative Language API but lack cachedContents never accumulate entries, since every creation attempt fails and gets negative-cached.

Metrics¶

Admin Endpoints¶

GET /admin/gemini-context-cache/stats¶

Returns the enabled flag, an effective-configuration echo, the current entry count, negative_entries, in_flight creations, a per-backend breakdown, and the hit/miss/ineligible/negative/creates/create_failures/evictions/cached_tokens counters.

POST /admin/gemini-context-cache/clear¶

Clears the router-side entry map and issues a best-effort remote DELETE for every tracked cachedContents resource.

Example response:

{ "success": true, "cleared_entries": 4 }

When the feature is disabled:

{ "success": false, "error": "Gemini context cache is not enabled" }

Configuration Reference¶

Tier 1: Prefix-Aware Routing¶

prefix_routing:
  enabled: true

  # Maximum bytes of prompt content used in prefix hash (default: 1024)
  max_prefix_length: 1024

  # CHWBL load cap epsilon: backend can handle (1 + epsilon) * avg_load
  # Range: 0.01 to 10.0 (default: 0.25)
  load_factor_epsilon: 0.25

  # Virtual nodes per backend on the consistent hash ring (default: 150)
  # Higher values improve distribution uniformity
  virtual_nodes: 150

  # Inject Anthropic cache_control markers into system prompts (default: false)
  anthropic_cache_control_injection: false

Tier 2: Response Cache¶

response_cache:
  enabled: true

  # Cache backend: "memory" (default) or "redis"
  # Changing backend requires restart; other fields support hot-reload
  backend: memory

  # Maximum cached entries before LRU eviction (default: 1000)
  capacity: 1000

  # TTL for cached entries (default: "5m")
  ttl: "5m"

  # Maximum response body size eligible for caching (default: 1 MiB)
  max_response_size: 1048576

  # Maximum streaming buffer size eligible for caching (default: 10 MiB)
  max_stream_buffer_size: 10485760

  # Semantic (similarity) cache, off by default. Serving also needs
  # control_plane.optimization.semantic_cache_enabled: true and the hub
  # policy's semantic_cache flag.
  semantic:
    enabled: true
    backend: embedder                 # configured backend serving the embedding model (required)
    embedding_model: bge-m3           # model requested from that backend (required)
    threshold: 0.92                   # cosine similarity; absent = hub semantic_similarity_bps
    max_entries: 1000                 # vectors held in the per-router index, 1..=1000000
    ttl: "5m"                         # entry TTL; absent = response_cache.ttl
    embedding_timeout_ms: 2000        # embedding call bound, 100..=60000; expiry is a miss
    max_input_bytes: 8192             # longer final user messages skip the path, 1..=1048576

Tier 3: Shared External Cache (Redis backend)¶

response_cache:
  enabled: true
  backend: redis

  redis:
    # Redis/Valkey connection URL
    url: "redis://redis:6379"

    # rediss:// for TLS; or set tls: true with redis:// URL
    # url: "rediss://redis:6380"

    # Connection pool size (default: 8)
    pool_size: 8

    # Key namespace prefix (must not contain glob characters)
    key_prefix: "cr:resp:"

    # Connection timeout in milliseconds (default: 3000)
    connect_timeout_ms: 3000

    # Per-command timeout in milliseconds (default: 1000)
    command_timeout_ms: 1000

    # Fallback capacity for in-memory cache when Redis is unreachable (default: 1000)
    fallback_capacity: 1000

    # TTL for fallback in-memory entries in seconds (default: 300)
    fallback_ttl_seconds: 300

    # Fall back to in-memory cache on Redis failure (default: true)
    fallback_to_memory: true

Tier 4: KV Cache Index¶

kv_cache_index:
  enabled: true

  # Index backend: "memory" (default) or "redis"
  # When "redis", reuses the connection pool from response_cache.redis
  backend: memory

  # Maximum prefix hash entries tracked (default: 100000)
  # Range: 100 to 10,000,000
  max_entries: 100000

  # TTL for index entries in seconds (default: 600)
  # Range: 1 to 86400
  entry_ttl_seconds: 600

  # Backend scoring weights (overlap + load + health must sum to 1.0)
  scoring:
    overlap_weight: 0.6
    load_weight: 0.3
    health_weight: 0.1

    # Minimum best-overlap to activate KV-aware routing (default: 0.3)
    # If no backend exceeds this, falls back to configured strategy
    min_overlap_threshold: 0.3

    # Tier weight multipliers (independent of the three main weights)
    # Applied to the raw overlap score before the weighted sum
    gpu_tier_weight: 1.0       # Multiplier for GpuHot (GPU-resident) data
    storage_tier_weight: 0.6   # Multiplier for StorageWarm (offloaded) data

  # Tiered storage awareness (GPU hot vs. external storage warm)
  storage_offloading:
    enabled: false               # Enable storage tier tracking (default: false)
    treat_eviction_as_offload: true  # Treat cache_evicted as offload to warm (default: true)

  # continuum-kv-listener SSE streams to subscribe to for KV cache events
  event_sources:
    - backend_name: vllm-1
      endpoint: "http://kv-listener.internal:7817/events/vllm-1"
      reconnect_interval_ms: 5000

    - backend_name: vllm-2
      endpoint: "http://kv-listener.internal:7817/events/vllm-2"
      reconnect_interval_ms: 5000

Supported endpoint schemes for event_sources[].endpoint: http, https, ws, wss. A source may instead set transport: event_bus (default sse) and omit endpoint; see Push Mode.

Metrics¶

All KV cache metrics use the prefix continuum_. Label values are sanitized against allowlists to prevent cardinality explosion.

Tier 1: Prefix Routing¶

continuum_prefix_routing_requests_total is recorded once per selection and only when the PrefixAwareHash strategy arm ran, so under any other selection_strategy it stays at zero by design. See Routing decision labels for what each label means in terms of the ring walk.

Example PromQL:

# Overflow rate (CHWBL load cap activations)
rate(continuum_prefix_routing_requests_total{strategy="overflow"}[5m])
/ rate(continuum_prefix_routing_requests_total[5m])

# Per-backend load distribution
continuum_prefix_routing_backend_distribution

Tier 2: Response Cache¶

Example PromQL:

# Cache hit rate over 5 minutes
rate(continuum_response_cache_requests_total{result="hit"}[5m])
/ rate(continuum_response_cache_requests_total{result=~"hit|miss"}[5m])

# Cache bypass rate (non-deterministic requests)
rate(continuum_response_cache_requests_total{result="skip"}[5m])

Tier 3: Redis Backend¶

Example PromQL:

# Redis P99 GET latency
histogram_quantile(0.99,
  rate(continuum_cache_redis_latency_seconds_bucket{operation="get"}[5m])
)

# Redis error rate
rate(continuum_cache_redis_errors_total[5m])

# Alert: fallback active
continuum_cache_fallback_active == 1

Tier 4: KV Cache Index¶

continuum_kv_index_routing_decisions_total increments once per scored selection: decision="kv_aware" when the scorer selected the backend, and decision="fallback" when the scorer evaluated a prefix key but no backend cleared the 0.3 combined-score threshold. Selections where the scorer did not run (because no scorer was registered or no prefix key was present) do not increment this counter. continuum_kv_index_overlap_score records the winning combined score (the weighted sum of overlap, load, and health) for kv_aware selections.

Example PromQL:

# KV-aware routing activation rate
rate(continuum_kv_index_routing_decisions_total{decision="kv_aware"}[5m])
/ rate(continuum_kv_index_routing_decisions_total[5m])

# Event drop rate per backend (indicates backpressure)
rate(continuum_kv_event_dropped_total[5m])

# P50 overlap score for routed requests
histogram_quantile(0.50, rate(continuum_kv_index_overlap_score_bucket[5m]))

# Disconnected event consumers
continuum_kv_consumer_connected == 0

Admin Endpoints¶

All admin endpoints are under the /admin prefix and require authentication if configured.

Prefix Routing¶

GET /admin/prefix-routing/stats¶

Returns prefix routing statistics including routing decision counts, overflow rate, backend load distribution, and CHWBL configuration.

Example response:

{
  "enabled": true,
  "config": {
    "max_prefix_length": 1024,
    "load_factor_epsilon": 0.25,
    "virtual_nodes": 150,
    "anthropic_cache_control_injection": false
  },
  "routing_decisions": {
    "total": 4926,
    "prefix_hash": 4821,
    "overflow": 93,
    "fallback": 12,
    "overflow_rate": "0.0189"
  },
  "backend_distribution": [
    { "backend": "vllm-1", "in_flight_requests": 4 },
    { "backend": "vllm-2", "in_flight_requests": 3 }
  ],
  "unique_prefixes": 247
}

Response Cache¶

GET /admin/response-cache/stats¶

Returns response cache statistics: hit/miss/skip counts, hit rate, entry count, memory usage, and Redis connection info (when applicable).

Example response:

{
  "enabled": true,
  "backend_type": "redis",
  "entries": 1243,
  "capacity": 5000,
  "requests": {
    "hit": 8912,
    "miss": 2341,
    "skip": 441,
    "total": 11694
  },
  "hit_rate": "0.7924",
  "evictions": 0,
  "size_bytes": 0,
  "config": {
    "backend": "redis",
    "ttl": "30m",
    "capacity": 5000,
    "max_response_size": 1048576,
    "max_stream_buffer_size": 10485760
  },
  "redis": {
    "connections": {
      "active": 3,
      "idle": 5
    },
    "fallback_active": false,
    "errors": {
      "connection": 0,
      "timeout": 2,
      "other": 0
    }
  }
}

POST /admin/response-cache/invalidate¶

Invalidates cached responses. Accepts a JSON body:

{ "clear_all": true }

Example response:

{
  "success": true,
  "action": "clear_all",
  "cleared_entries": 1243
}

KV Cache Index¶

GET /admin/kv-index/stats¶

Returns KV cache index statistics: entry counts, routing decision breakdown, query latency counters, and overlap score counts.

Example response:

{
  "enabled": true,
  "config": {
    "backend": "memory",
    "max_entries": 100000,
    "entry_ttl_seconds": 600,
    "event_sources_count": 2,
    "scoring": {
      "overlap_weight": 0.6,
      "load_weight": 0.3,
      "health_weight": 0.1,
      "min_overlap_threshold": 0.3
    }
  },
  "index": {
    "prefix_count": 312,
    "entry_count": 618,
    "total_hits": 12490,
    "total_evictions": 83
  },
  "event_sources": [
    {
      "backend_name": "vllm-1",
      "connected": true,
      "events_received": 8412,
      "events_dropped": 0,
      "last_event_at": "2026-03-13T10:24:17Z",
      "reconnect_count": 0
    }
  ],
  "routing_decisions": {
    "kv_aware": 9841,
    "fallback": 2649,
    "total": 12490
  },
  "query_latency_count": 12490,
  "overlap_score_count": 9841
}

GET /admin/kv-index/backends¶

Returns per-backend KV cache event statistics: events received, processed, dropped, connection status, and index event counts (created/evicted).

Example response:

{
  "enabled": true,
  "backends": [
    {
      "backend_name": "vllm-1",
      "connection": {
        "connected": true,
        "reconnect_count": 0,
        "last_event_at": "2026-03-13T10:24:17Z"
      },
      "events": {
        "received": 8412,
        "dropped": 0,
        "index_created": 7981,
        "index_evicted": 431
      }
    }
  ]
}

POST /admin/kv-index/clear¶

Clears all entries from the KV cache index. The index rebuilds automatically from incoming events. Intended for debugging.

Example response:

{
  "success": true,
  "entries_before_clear": 618,
  "cleared_entries": 2
}

Deployment Guide¶

Tier 1 Only (Minimal Configuration)¶

Enable prefix routing for GPU KV cache locality without any external dependencies:

prefix_routing:
  enabled: true
  load_factor_epsilon: 0.25
  virtual_nodes: 150

Effective when multiple instances of the same model run across backends and system prompts are long (>128 tokens).

Tier 1 + 2 (Response Cache)¶

Add response caching to eliminate repeated deterministic requests entirely:

prefix_routing:
  enabled: true

response_cache:
  enabled: true
  backend: memory
  capacity: 5000
  ttl: "10m"

Effective when the application makes repeated identical requests (e.g., document QA with fixed system prompts and fixed queries).

Tier 1 + 2 + 3 (Distributed Response Cache)¶

For multi-instance deployments, share the response cache across all router instances:

prefix_routing:
  enabled: true

response_cache:
  enabled: true
  backend: redis
  ttl: "30m"
  redis:
    url: "redis://redis-service:6379"
    pool_size: 16
    key_prefix: "cr:resp:"
    fallback_to_memory: true
    fallback_capacity: 2000

Redis/Valkey must be accessible from all router pods. Use rediss:// URL or tls: true for encrypted connections.

All Tiers (Full KV-Aware Routing)¶

Enable all tiers for maximum GPU cache reuse:

prefix_routing:
  enabled: true
  load_factor_epsilon: 0.20

response_cache:
  enabled: true
  backend: redis
  ttl: "30m"
  redis:
    url: "redis://redis-service:6379"
    pool_size: 16

kv_cache_index:
  enabled: true
  backend: redis   # shares the pool from response_cache.redis
  max_entries: 500000
  entry_ttl_seconds: 900
  scoring:
    overlap_weight: 0.6
    load_weight: 0.3
    health_weight: 0.1
    min_overlap_threshold: 0.25
    gpu_tier_weight: 1.0       # GPU-resident data gets full overlap credit
    storage_tier_weight: 0.6   # Offloaded data is still valuable but discounted
  storage_offloading:
    enabled: true              # Track GPU hot vs. storage warm tiers
    treat_eviction_as_offload: true
  event_sources:
    - backend_name: vllm-1
      endpoint: "http://kv-listener.internal:7817/events/vllm-1"
    - backend_name: vllm-2
      endpoint: "http://kv-listener.internal:7817/events/vllm-2"

Engine Event Source Requirements¶

Start continuum-kv-listener on the internal network and configure each engine to publish KV events to its ZMQ endpoint. The router consumes only the listener's SSE endpoint through kv_cache_index.event_sources[], so engine containers do not need to expose an extra HTTP producer port.

continuum-kv-listener --config config.kv-listener.yaml

• vLLM and SGLang: vLLM does not serve an HTTP /v1/kv_events route; it publishes msgpack KVEventBatch frames over a ZMQ PUB socket when started with --kv-events-config (for example, {"publisher":"zmq","endpoint":"tcp://kv-listener.internal:5557","topic":""}). SGLang uses the same ZMQ event format.
• Prefix reconstruction: Because engine block hashes are internal digests rather than the router's SHA-256 prefix key, continuum-kv-listener reconstructs the prompt prefix and computes the key with extract_prefix_key. Keep the ZMQ bind address on an internal network, and use push mode if the router cannot reach the listener directly.
• Native TensorRT-LLM: Set engine: trtllm on the listener source and point trtllm_endpoint at the trtllm-serve /kv_cache_events URL, with block reuse and a positive event buffer size enabled in TensorRT-LLM. The URL host must be listed in allowed_publishers, redirects are rejected, and the poll interval, request timeout, and response size are all bounded.
• TensorRT-LLM cache salt requirement: Because trtllm-serve does not expose a public /detokenize endpoint, native TensorRT-LLM sources rely on router-issued cache salts. The first stored block in a chain must carry a salt matching the source backend; unsalted or mismatched chains are discarded. The listener never logs or publishes salts, raw response bodies, or token IDs.

Redis/Valkey Sizing¶

For response cache sizing, estimate:

• Average serialized response size: 2–10 KB per entry
• capacity = (target_hit_rate * rps * avg_unique_rate) / eviction_frequency

For the KV index:

• Each (prefix, backend) entry consumes approximately 200 bytes in memory
• Set max_entries to at least num_unique_prefixes * num_backends * 2 for headroom

High Availability Considerations¶

• The response cache and KV index tolerate Redis failure via automatic in-memory fallback (Tier 3).
• The KV index rebuilds from the SSE streams on restart; no persistence is required.
• Prefix routing (Tier 1) has no external dependencies and is always available.
• Deploy Redis with replication (Sentinel or Cluster) if cache persistence across Redis restarts is required.

Performance Characteristics¶

Tier 1: Prefix Routing¶

• Prefix key extraction (SHA256): < 10 µs per request
• CHWBL ring lookup: O(log N) where N = virtual_nodes * num_backends; < 5 µs for typical deployments
• No network I/O; operates entirely in-process

Tier 2: Response Cache¶

• In-memory cache lookup: < 1 µs
• Cache key computation (SHA256): < 5 µs
• Cache hit serves the entire response without any backend latency
• Semantic lookup: one embedding round trip to the pinned backend plus a partition-local scan of at most max_entries normalized vectors

Tier 3: Redis Backend¶

• Redis GET latency (LAN): 0.1–2 ms typical; P99 < 5 ms
• Redis SET latency: similar to GET
• Command timeout default: 1 second; operations exceeding this activate fallback

Tier 4: KV Cache Index¶

• InMemoryKvIndex.query_backends(): < 100 µs (DashMap read, no allocation on empty result)
• RedisKvIndex.query_backends(): same as Redis GET latency (0.1–2 ms)
• KvOverlapScorer.prepare(): one query_backends() call per unique prefix per 100 ms window
• KvOverlapScorer.score(): < 1 µs (synchronous read from pre-fetched cache)
• 1000 scoring calls: < 100 ms total (verified by unit benchmark in scorer.rs)

Expected Gains¶

The following are illustrative estimates based on typical LLM workload patterns:

Actual gains depend on workload prefix overlap, GPU memory capacity, and backend configuration.