KV Cache Optimization¶

Continuum Router implements a four-tier KV cache optimization system that reduces redundant computation on LLM backends. The tiers are independent toggles rather than layers stacked on one another: each has its own configuration section, and whichever ones are enabled are consulted in a fixed order (Tier 2 response cache, then the Tier 4 scorer, then the configured selection_strategy, which is Tier 1 only when it is set to PrefixAwareHash).

Table of Contents¶

• Overview
• Four-Tier Caching Strategy
• Tier 1: Prefix-Aware Sticky Routing
• Tier 2: Response Cache
• Tier 3: Shared External Cache
• Tier 4: Backend KV Cache Index
• Backend Selection Pipeline
• Gemini Context Cache
• Configuration Reference
• Metrics
• Admin Endpoints
• Deployment Guide
• Performance Characteristics

Overview¶

Modern LLM inference engines (vLLM, TensorRT-LLM, SGLang) maintain a KV cache in GPU memory that stores attention key-value tensors computed for the token prefix of a request. When the same prefix is seen again, the engine can skip recomputing those tensors, a substantial GPU time saving for long system prompts or repeated context.

Continuum Router maximizes KV cache reuse across backends through four complementary mechanisms:

1. Prefix-Aware Sticky Routing — routes requests sharing the same prompt prefix to the same backend via consistent hashing, keeping GPU KV cache warm.
2. Response Cache — serves repeated deterministic requests directly from router memory or Redis without hitting any backend at all.
3. Shared External Cache — stores response cache state in Redis/Valkey so multiple router instances share the same cache entries.
4. Backend KV Cache Index — tracks which backends actually hold GPU-resident KV tensors for recent prefixes, enabling fine-grained routing decisions informed by real cache state.

Four-Tier Caching Strategy¶

flowchart TD
    Client([Client Request])
    RC{Response\nCache hit?}
    PRL{Prefix key\navailable?}
    REG{KV index scorer\nregistered?}
    SCORE[Composite Score\noverlap + load + health]
    THRESH{Best combined\nscore > 0.3?}
    STRATEGY[Configured selection_strategy]
    PAH{PrefixAwareHash:\nprefix key present?}
    CHWBL[CHWBL Hash Ring\nPrefix → Backend, load cap]
    MHASH[Model-name consistent hash\nno load cap]
    BACKEND([Selected Backend])
    STORE_RC[Store Response\nin Cache]

    Client --> RC
    RC -->|HIT| Client
    RC -->|MISS| PRL
    PRL -->|Yes| REG
    REG -->|Yes| SCORE
    SCORE --> THRESH
    THRESH -->|Yes| BACKEND
    THRESH -->|No| STRATEGY
    REG -->|No| STRATEGY
    PRL -->|No| STRATEGY
    STRATEGY -->|PrefixAwareHash| PAH
    STRATEGY -->|any other strategy| BACKEND
    PAH -->|Yes| CHWBL
    PAH -->|No| MHASH
    CHWBL --> BACKEND
    MHASH --> BACKEND
    BACKEND --> STORE_RC
    STORE_RC --> Client

The tiers are not mutually exclusive, and they are not stacked. Tier 2 intercepts the entire request before any backend is contacted, and Tier 3 is only the storage substrate underneath it. Tier 4 runs first inside backend selection, and it is gated on prefix_routing.enabled (which produces the prefix key) plus kv_cache_index.enabled (which produces the index), never on selection_strategy. The configured strategy decides whenever the scorer does not, and that strategy is Tier 1 only when it is PrefixAwareHash.

Tier 1: Prefix-Aware Sticky Routing¶

Tier 1 routes requests that share a common prompt prefix to the same backend, maximizing the probability that the GPU KV cache on that backend is already warm for those tokens.

Prefix Key Extraction¶

For each incoming chat completion request, the router extracts a prefix key: a 32-byte SHA256 digest that uniquely identifies the semantic anchor of the request.

The extraction logic handles both OpenAI and Anthropic request formats:

The hash is computed as:

# With system prompt:
SHA256(model_bytes ++ "\x00" ++ "S" ++ system_bytes[:max_prefix_length])

# Without system prompt (first message fallback):
SHA256(model_bytes ++ "\x00" ++ "M" ++ first_msg_bytes[:max_prefix_length])

The \x00 separator prevents length-extension collisions between the model name and content. The tag bytes S/M prevent identical text from hashing to the same value when it appears as a system prompt versus a first user message.

The max_prefix_length parameter (default: 1024 bytes) truncates the content before hashing, with UTF-8 boundary awareness to avoid splitting multibyte characters.

Implementation: src/core/prefix_key.rs, src/core/hashing.rs

Consistent Hashing with Bounded Loads (CHWBL)¶

The PrefixAwareHash selection strategy uses a consistent hash ring to map prefix keys to backends. Simple consistent hashing can produce uneven load distribution when some prefixes are far more popular than others. Continuum Router addresses this with the Consistent Hashing with Bounded Loads (CHWBL) algorithm.

CHWBL adds a load cap: a backend can handle at most (1 + epsilon) * average_load requests simultaneously. When a backend is at its load cap, the request overflows to the next node clockwise on the ring.

epsilon = 0.25  →  a backend can be at most 25% above average load

The ring is populated with virtual_nodes (default: 150) virtual replicas per backend to improve key distribution uniformity. virtual_nodes is applied to the live pool on hot-reload: changing it rebuilds the consistent-hash ring at the new size without a restart.

When no prefix key exists¶

The CHWBL ring above is reached only when the request actually carries a prefix key. Two cases produce no key:

• prefix_routing.enabled is false, which is the default. Both extractors, request_prefix_key and typed_request_prefix_key, return None immediately, so no request in the deployment ever has a key.
• prefix_routing.enabled is true, but the request body carries neither a system prompt nor any messages, so there is no anchor to hash.

In both cases the PrefixAwareHash strategy hashes the model name instead and walks the ring through the plain consistent-hash path. That walk has no bounded-load cap: it takes the first node clockwise from the model's hash and never overflows to a less busy backend. Every request for one model therefore lands on the same backend for as long as pool membership is unchanged, no matter how loaded that backend becomes.

PrefixAwareHash consequently requires prefix_routing.enabled: true to do anything prefix-related at all. With prefix routing disabled the strategy is plain model-name consistent hashing under a different name, and it is the one combination on this page that silently does less than it looks like it does. The router reports it: continuum-router config validate emits a warning at the selection_strategy path (the configuration stays valid), and the same message is logged at startup and on any hot reload that applies the combination.

Routing decision labels¶

continuum_prefix_routing_requests_total{strategy=...} is recorded once per selection, and only when the PrefixAwareHash arm actually ran. Other strategies record nothing on this metric family.

• prefix_hash: the prefix-hashed owner took the request. This covers the ordinary case (the owner was under the load cap) and the full-lap case where every backend was at the cap and the ring walk kept the owner to preserve prefix locality.
• overflow: the owner was at or over the load cap, so a later node clockwise on the ring took the request.
• fallback: the request carried no prefix key, so the model name was hashed and the ring was walked with no load cap. See When no prefix key exists above.

Anthropic Cache Control Injection¶

When anthropic_cache_control_injection: true is set, the router automatically adds cache_control: { type: "ephemeral" } markers to the system prompt of native-Anthropic requests. This activates Anthropic's server-side prompt caching, which is distinct from the router-level KV cache but complementary to it.

This flag is a fleet-wide master enable and is read per-request (hot-reloadable). Its precedence with the per-backend anthropic_auto_cache_control:

• Per-backend anthropic_auto_cache_control governs by default and is true when unset, so native-Anthropic backends inject automatically out of the box.
• prefix_routing.anthropic_cache_control_injection: true forces injection on even for a backend that set anthropic_auto_cache_control: false.
• Its default (false) is a strict no-op: the per-backend setting alone decides, exactly as before the flag was wired.
• Injection only ever touches native-Anthropic requests and never other providers. It targets the last system block only, and if the request already carries any cache_control (system, message, or tool), injection is skipped to preserve the caller's explicit caching intent.

Tier 2: Response Cache¶

The response cache stores complete LLM responses for deterministic requests, allowing the router to serve repeated queries without contacting any backend.

Cache Eligibility¶

A request is eligible for caching when all of the following are true:

• temperature is 0 or absent
• The request does not use streaming (or uses streaming with buffering enabled)
• The accumulated response size is within max_response_size

Requests with non-zero temperature are probabilistic and are never cached. The response header X-Cache: HIT, X-Cache: MISS, or X-Cache: BYPASS indicates the cache disposition.

Cache Key Computation¶

The cache key is a SHA256 hash over all parameters that affect LLM output:

SHA256(
    model,
    "\x00",
    SHA256(messages),   // pre-hashed messages array
    "\x00",
    temperature_bytes,
    "\x00",
    SOME/NONE_tag + max_tokens_bytes,
    "\x00",
    SOME/NONE_tag + top_p_bytes,
    "\x00",
    SOME/NONE_tag + tenant_id_bytes,
)

SOME/NONE tag bytes prevent collisions between None and Some(0) for optional parameters. The tenant ID is included to provide multi-tenant isolation: tenants cannot read each other's cached responses.

Implementation: src/infrastructure/cache/response_cache.rs

Streaming Cache¶

For streaming responses, the router accumulates the SSE stream into a buffer up to max_stream_buffer_size (default: 10 MiB). If the complete stream fits within the limit, the buffer is stored as a single serialized blob and replayed as a synthetic SSE stream on cache hits.

Cache Eviction¶

The in-memory backend uses LRU eviction when the entry count reaches capacity. The Redis backend relies on TTL-based expiration managed by Redis itself.

Semantic Cache¶

Behind the exact and prefix paths sits a third, opt-in path (issue #1571) that serves a deterministic request from an earlier response whose final user message is semantically close to the new one. It is consulted only after the exact and prefix lookups miss, and only when every one of these holds:

• response_cache.semantic.enabled: true, with backend and embedding_model pinned
• the router's own control_plane.optimization.semantic_cache_enabled: true
• the hub optimization policy's semantic_cache (together with exact_cache) for the org, on a tier with cache_enabled
• a non-streaming /v1/chat/completions request with temperature: 0 whose final message is a user turn carrying text of at most max_input_bytes

The lookup runs after the guardrail input gate, so a prompt the gate blocks is never sent to the embedder and a transform verdict is applied before the text is embedded. Each eligible request then costs exactly one POST /v1/embeddings call to the pinned backend, bounded by embedding_timeout_ms; on a miss the router keeps that vector and indexes the backend's response under it once the response arrives. A failed or timed-out embedding call is a plain miss and never fails the chat request. The pin follows the pinned-backend visibility rule shared with the LLM classifier and the guardrail transport: an internal: true backend is reachable because the operator named it, an enabled: false backend is refused because prompt text would be sent to it.

Partitioning is the security boundary of this path. Similarity is computed only among entries that share a partition digest, and the digest folds in the caller's cache identity (the route-scoped local identity plus the hub key id), the model, the embedding model, every response-shaping field of the request (tools, response format, sampling bounds, seed), and the entire conversation preceding the final user turn, system prompt included. Two API keys, two system prompts, or two tool sets therefore never share an entry.

A candidate is served when its cosine similarity reaches threshold, or the hub policy's semantic_similarity_bps when threshold is unset. The response carries X-Cache: HIT and X-Cache-Mode: semantic, and the hub usage record is stamped cache_hit_type = semantic; like every local hit it is metadata-only and charges neither token windows nor budgets. Bodies live in the response cache store under the sem: keyspace, so the store's capacity and TTL bound them; the vector index is a bounded, per-router structure (max_entries, oldest-first eviction) with no external dependency. Entry TTL is the hub policy's cache_ttl_secs when directed, else semantic.ttl, else response_cache.ttl. A shared or cross-router index, external vector databases, and streaming semantic hits are out of scope.

Semantic bodies share response_cache.capacity with the exact and prefix keyspaces, so a busy semantic path adds real eviction pressure there too; size capacity for the combined volume, or keep semantic.max_entries low enough that semantic bodies stay a small share of the store.

Implementation: src/infrastructure/cache/semantic/ (partition, index, embedding shapes) and src/proxy/semantic_cache.rs (serving path).

Tier 3: Shared External Cache¶

The shared external cache provides a CacheStore trait abstraction over Redis/Valkey, allowing multiple router instances to share response cache state.

CacheStore Trait¶

pub trait CacheStore: Send + Sync + 'static {
    async fn get(&self, key: &str) -> CacheStoreResult<Option<Vec<u8>>>;
    async fn set(&self, key: &str, value: &[u8], ttl: Duration) -> CacheStoreResult<()>;
    async fn delete(&self, key: &str) -> CacheStoreResult<()>;
    async fn clear(&self) -> CacheStoreResult<()>;
    async fn stats(&self) -> CacheStoreStats;
}

Implementations: InMemoryCacheStore (default, LRU + TTL), RedisCacheStore (Redis/Valkey with connection pooling).

Implementation: src/infrastructure/cache/store.rs

Redis Backend¶

RedisCacheStore uses deadpool-redis for connection pooling. All keys are namespaced with a configurable prefix (default: cr:resp:) to avoid collisions with other applications sharing the same Redis instance.

Key namespacing format:

cr:resp:<sha256-cache-key-hex>

Operations use SET EX for writes and GET for reads, with configurable command timeouts (default: 1s).

Automatic Fallback¶

When Redis is unreachable, RedisCacheStore transparently activates an in-memory fallback cache. The fallback is activated on the first connection failure and a background health-monitor task (running every 30 seconds) attempts to restore the Redis connection. On recovery, the flag is cleared and subsequent operations go back to Redis.

The continuum_cache_fallback_active metric (value 1) indicates that fallback mode is currently active.

Connection Pool Sharing¶

The deadpool_redis::Pool is stored as an Arc in AppState and shared between the response cache and the KV cache index (Tier 4). This avoids double-counting connections and simplifies configuration: both consumers reuse the same pool credentials.

Tier 4: Backend KV Cache Index¶

Tier 4 tracks in real time which backends hold GPU-resident KV tensors for specific token prefix hashes. This enables routing decisions based on actual GPU cache state rather than statistical affinity.

Event Consumption¶

The supported producer bridge is the continuum-kv-listener binary. It receives vLLM and SGLang msgpack records over ZMQ or polls native trtllm-serve JSON events over HTTP, resolves block chains to the same prefix hash as the router, and exposes a router-compatible SSE stream per backend (for example http://kv-listener.internal:7817/events/vllm-1). KvEventConsumerManager spawns a background Tokio task per backend that subscribes to the listener stream and processes events.

A TensorRT-LLM created event resets the listener's tracked block state, stored events create or extend chains through the shared source processor, and removed events evict their resolved prefixes. The poller uses the event ID to reject duplicate responses and clear stale state after a gap. Dynamo deployments that already publish the vLLM-compatible ZMQ event format should continue to use the ZMQ source instead of configuring a second native HTTP source.

Event types:

Each event carries a prefix_hash (hex string) and an optional token_count indicating how many tokens are cached for that prefix on that backend.

SSE parsing details:

• The consumer preferentially uses the SSE event: field to determine event type; the JSON event field is a fallback.
• The buffer is capped at 1 MiB (MAX_SSE_BUFFER_SIZE) to protect against malformed streams.
• On connection failure or stream end, the consumer applies exponential backoff (initial: 1s, max: 60s) before reconnecting.

Implementation: src/infrastructure/kv_index/event_consumer.rs

Producer bridge implementation: crates/continuum-kv-listener/

Index Structure¶

Events are fed into a KvCacheIndex implementation, which maintains a mapping:

prefix_hash → {backend_id → token_count}

The token count acts as the score: a backend holding 2048 cached tokens for a prefix ranks higher than one holding 512.

Two implementations are provided:

InMemoryKvIndex¶

• DashMap<String, PrefixEntry> for lock-free concurrent reads
• LRU eviction when entry count reaches max_entries (evicts the oldest 10% of entries)
• TTL-based expiration checked lazily on query_backends()
• Periodic cleanup_expired() removes stale entries proactively
• Default: 100,000 max entries, 300s TTL

RedisKvIndex¶

• Stores each prefix as a Redis sorted set: ZADD cr:kvidx:<prefix_hash> <token_count> <backend_id>
• EXPIRE is pipelined with ZADD in a single atomic round-trip
• ZREVRANGEBYSCORE +inf -inf WITHSCORES returns backends ranked by descending score
• Enables sharing of KV index state across multiple router instances
• Key prefix: cr:kvidx:

Implementation: src/infrastructure/kv_index/index.rs

Storage Tier Awareness¶

When storage_offloading.enabled is true, the index tracks two storage tiers for each (prefix, backend) entry:

Tier transitions from events:

The treat_eviction_as_offload option controls whether generic cache_evicted events (which do not carry tier information) are treated as offloads to warm storage or as permanent removals. This is useful when vLLM backends emit only cache_created and cache_evicted events without the explicit cache_offloaded event type.

Implementation: src/infrastructure/kv_index/types.rs

Overlap Scoring¶

The KvOverlapScorer implements the BackendScorer trait and computes a composite score for each backend that holds cached data for the request's prefix:

final_score = overlap_weight   * (raw_overlap * tier_multiplier)
            + load_weight      * (1.0 - load_ratio)
            + health_weight    * health_score

Where:

• raw_overlap = backend_token_count / max_token_count_across_backends (0.0 to 1.0)
• tier_multiplier = gpu_tier_weight for GpuHot data, or storage_tier_weight for StorageWarm data
• load_ratio = backend_in_flight / max_in_flight_across_backends (0.0 to 1.0)
• health_score = backend_success_rate (0.0 to 1.0)

A backend with no entry in the index for that prefix scores 0.0 and is not ranked at all, so load and health only ever separate backends that actually hold the data. Without that rule an idle, healthy backend holding nothing would score load_weight + health_weight, which is 0.4 on the defaults, clear the pool's combined-score threshold of 0.3, and override the configured strategy for every request on that prefix.

Where the weights come from¶

The six values are read from the kv_cache_index.scoring block and carried into the scorer when it is registered at startup. The effective weights are logged once at INFO at that point, so an operator can confirm a non-default block took effect. kv_cache_index is a restart-required section: editing scoring and hot-reloading the config does not rebuild the scorer.

Default weights:

The three main weights (overlap_weight, load_weight, health_weight) must sum to exactly 1.0 (validated at load).

The tier weight multipliers (gpu_tier_weight, storage_tier_weight) are independent: they scale the raw overlap score before the weighted sum is computed. A StorageWarm backend with 100 cached tokens scores lower than a GpuHot backend with 100 cached tokens because the tier multiplier reduces the effective overlap signal.

Load and health inputs¶

The load term comes from the backend pool's in-flight tracker, the same live gauge the CHWBL load cap reads, not from the BackendStats.in_flight_requests field (which the shared stats map no longer maintains). prepare() takes one snapshot of the tracker alongside the index query and stores it with the cached result, so every backend in a scoring pass is ranked against a single coherent view of the load rather than a gauge that moves between reads. Staleness is bounded by the same 100 ms result TTL described below.

The health term is the backend's success rate from the shared stats map, read synchronously; when that lock is momentarily contended, the scorer substitutes a neutral 1.0 rather than blocking.

Minimum overlap threshold¶

min_overlap_threshold (default: 0.3) gates per holder, on the tier-weighted normalized overlap (raw_overlap * tier_multiplier). A holder below the threshold is treated exactly like a backend that holds nothing and scores 0.0.

Two consequences follow from the normalization. The best GPU-resident holder has a raw overlap of 1.0 by construction, so its tier-weighted value is exactly gpu_tier_weight and it passes whenever gpu_tier_weight >= min_overlap_threshold, which holds comfortably on the defaults (1.0 against 0.3). A backend whose only copy of the prefix has been offloaded to warm storage passes only when storage_tier_weight >= min_overlap_threshold, which is what the tier multiplier is for: on the defaults 0.6 clears 0.3, but lowering storage_tier_weight below the threshold turns warm-only copies into non-holders.

The loader does not tie the tier weights to the threshold (both weights are only checked for being non-negative), so gpu_tier_weight below min_overlap_threshold still loads cleanly and produces a scorer in which no backend ever passes the gate and the configured strategy always decides. Because that combination is indistinguishable from a healthy one at runtime, continuum-router config validate reports it as a warning at kv_cache_index.scoring.gpu_tier_weight and startup logs the same message once. The warning is raised only when the scorer would actually be registered, that is when prefix_routing.enabled and kv_cache_index.enabled both hold. Keep gpu_tier_weight >= min_overlap_threshold unless turning scored selection off is the intent.

When no eligible backend passes the gate, every score is 0.0, nothing exceeds the pool's combined-score threshold, and the configured selection_strategy decides. That combined-score threshold is 0.3, compiled into the pool as DEFAULT_SCORER_THRESHOLD; no configuration field sets it. The comparison is strict, so a selection needs a combined score above 0.3, not equal to it.

Async preparation model¶

KvCacheIndex.query_backends() is async but BackendScorer.score() must be synchronous. The scorer uses a two-phase design: prepare() fetches the index query result and the in-flight snapshot and caches both (with a 100 ms internal TTL), then score() reads synchronously from that cache.

Implementation: src/infrastructure/kv_index/scorer.rs

Backend Selection Pipeline¶

When a chat completion request arrives, the router executes the following pipeline:

sequenceDiagram
    participant C as Client
    participant R as Router
    participant RC as Response Cache
    participant PK as Prefix Key
    participant KVI as KV Index
    participant CHWBL as CHWBL Ring
    participant B as Backend

    C->>R: POST /v1/chat/completions
    R->>RC: Lookup cache key (T=0 only)
    alt Cache HIT
        RC-->>R: Cached response
        R-->>C: 200 OK (X-Cache: HIT)
    else Cache MISS
        R->>PK: Extract prefix key
        alt Prefix key available and KV index scorer registered
            R->>KVI: prepare(prefix_hash)
            KVI-->>R: Backend scores from index
            R->>R: Score = overlap + load + health (holders only)
            alt Best combined score > 0.3
                R->>B: Forward request (KV-aware routing)
            else At or below threshold
                R->>R: Run configured selection_strategy
                R->>CHWBL: PrefixAwareHash: map prefix → backend (load cap)
                R->>B: Forward request
            end
        else Prefix key available, no scorer registered
            R->>R: Run configured selection_strategy
            R->>CHWBL: PrefixAwareHash: map prefix → backend (load cap)
            R->>B: Forward request
        else No prefix key
            R->>R: Run configured selection_strategy
            R->>CHWBL: PrefixAwareHash without a prefix key: hash the model name (no load cap)
            R->>B: Forward request
        end
        B-->>R: Response
        R->>RC: Store response (T=0)
        R-->>C: 200 OK (X-Cache: MISS)
    end

The KV overlap scorer runs before the strategy rather than after it, and it composes with prefix-aware routing rather than replacing it. When at least one eligible backend holds data for the prefix and the best combined score clears the pool's compiled-in threshold of 0.3, the scorer decides. Otherwise the configured selection_strategy decides, and the CHWBL ring is that strategy only when selection_strategy is PrefixAwareHash; under RoundRobin, WeightedRoundRobin, LeastLatency, Random, or ConsistentHash the below-threshold path runs that strategy instead.

Tier 4 is gated on prefix_routing.enabled (which produces the prefix key the scorer needs) plus kv_cache_index.enabled (which produces the index), never on selection_strategy. RoundRobin with both of those enabled therefore gives fully KV-aware selection, with round-robin serving only as the below-threshold fallback.

Gemini Context Cache¶

Tiers 1 through 4 target backends with a GPU-resident KV cache (vLLM, TensorRT-LLM, SGLang). Google's Generative Language API has no equivalent event stream to observe, so type: gemini backends get a separate, opt-in mechanism instead: the router manages Google's server-side cachedContents resource on their behalf.

Overview¶

When enabled, the router caches large, stable system-prompt prefixes sent to type: gemini backends as cachedContents resources and reuses them across requests. This is transparent to OpenAI-compatible clients: they keep sending the same plain OpenAI-compatible request, and the router rewrites it before forwarding.

The compat request shape references a cache through extra_body.google.cached_content = "cachedContents/..."; the native generateContent shape instead carries a top-level cachedContent field and drops systemInstruction entirely, since the instruction now lives in the cached resource.

Google bills cached tokens at a reduced input rate plus a storage fee per token-hour, and it enforces a per-model minimum cacheable token count (roughly 2,048 tokens for Gemini 2.5-class models and 4,096 for 3.x-class models). A cache only pays off when a prefix is both large and reused often enough to outrun the storage fee, which is why the feature defaults to enabled: false and keeps ttl short. The router does not tokenize requests; it approximates the token-count floor with a conservative byte heuristic (min_prefix_bytes) and negative-caches any prefix Google rejects, so an undersized prefix is never retried more than once per negative_ttl_seconds window.

With gemini_context_cache.enabled: false (the default), the feature is a strict no-op: no map is allocated, no background task runs, and no per-request latency is added.

Request Lifecycle¶

A request is eligible when it targets a type: gemini backend and carries a system prompt of at least min_prefix_bytes. Eligible requests follow this path:

1. Digest the prefix. The router computes a digest over the model name and the full system prompt, mirroring the hashing in src/core/prefix_key.rs so prefix-routing stickiness and the context-cache lookup share one identity. Unlike the Tier 1 prefix key, this digest covers the entire system text with no truncation.
2. Map hit (entry not expired): the router injects the cached-content reference and drops the duplicated system prompt from the outgoing request. When extend_on_hit is set and the entry is within 20% of its TTL from expiry, the router PATCHes the TTL forward in the background.
3. Map miss (not negative-cached): the router forwards the request unmodified right away and spawns a single-flight background creation (POST /v1beta/cachedContents with model, systemInstruction, and ttl). The first request for a given prefix never pays creation latency; only later requests sharing the same prefix receive the injection.
4. Creation rejected with HTTP 400 (typically because the prefix is below the model's minimum cacheable token count): the router negative-caches the prefix for negative_ttl_seconds, so exactly one creation attempt is made per window.
5. Eviction. Entries beyond max_entries are LRU-evicted. Both eviction and the admin clear endpoint issue a best-effort DELETE against Google so storage billing stops early; Google's own TTL expiry remains the backstop. The negative cache shares the same max_entries cap: when it is full, expired rejection windows are swept first, and a new rejection is dropped only if the map is still full afterward.

Configuration Reference¶

gemini_context_cache:
  enabled: false

  # Minimum system-prompt size (UTF-8 bytes) to attempt caching (default: 16384)
  # Must be >= 1024. The router does not tokenize; this is a byte-based proxy
  # for Google's per-model minimum cacheable token count.
  min_prefix_bytes: 16384

  # TTL applied when creating a cachedContents resource (default: "10m")
  # Must parse as a duration and be >= 60s
  ttl: "10m"

  # Extend the TTL in the background when a hit lands within 20% of expiry (default: false)
  extend_on_hit: false

  # Maximum tracked entries before LRU eviction (default: 1000)
  # Range: 10 to 1,000,000
  max_entries: 1000

  # Timeout for the cachedContents creation call, in milliseconds (default: 5000)
  # Range: 500 to 60000
  create_timeout_ms: 5000

  # TTL for negative-caching a prefix Google rejected, in seconds (default: 600)
  # Must be >= 1
  negative_ttl_seconds: 600

Scope: only chat completions on type: gemini backends are covered. The /v1/responses Gemini converter, image generation, and embeddings paths do not participate. Proxies or self-hosted endpoints that mimic the Generative Language API but lack cachedContents never accumulate entries, since every creation attempt fails and gets negative-cached.

Metrics¶

Admin Endpoints¶

GET /admin/gemini-context-cache/stats¶

Returns the enabled flag, an effective-configuration echo, the current entry count, negative_entries, in_flight creations, a per-backend breakdown, and the hit/miss/ineligible/negative/creates/create_failures/evictions/cached_tokens counters.

POST /admin/gemini-context-cache/clear¶

Clears the router-side entry map and issues a best-effort remote DELETE for every tracked cachedContents resource.

Example response:

{ "success": true, "cleared_entries": 4 }

When the feature is disabled:

{ "success": false, "error": "Gemini context cache is not enabled" }

Configuration Reference¶

Tier 1: Prefix-Aware Routing¶

prefix_routing:
  enabled: true

  # Maximum bytes of prompt content used in prefix hash (default: 1024)
  max_prefix_length: 1024

  # CHWBL load cap epsilon: backend can handle (1 + epsilon) * avg_load
  # Range: 0.01 to 10.0 (default: 0.25)
  load_factor_epsilon: 0.25

  # Virtual nodes per backend on the consistent hash ring (default: 150)
  # Higher values improve distribution uniformity
  virtual_nodes: 150

  # Inject Anthropic cache_control markers into system prompts (default: false)
  anthropic_cache_control_injection: false

Tier 2: Response Cache¶

response_cache:
  enabled: true

  # Cache backend: "memory" (default) or "redis"
  # Changing backend requires restart; other fields support hot-reload
  backend: memory

  # Maximum cached entries before LRU eviction (default: 1000)
  capacity: 1000

  # TTL for cached entries (default: "5m")
  ttl: "5m"

  # Maximum response body size eligible for caching (default: 1 MiB)
  max_response_size: 1048576

  # Maximum streaming buffer size eligible for caching (default: 10 MiB)
  max_stream_buffer_size: 10485760

  # Semantic (similarity) cache, off by default. Serving also needs
  # control_plane.optimization.semantic_cache_enabled: true and the hub
  # policy's semantic_cache flag.
  semantic:
    enabled: true
    backend: embedder                 # configured backend serving the embedding model (required)
    embedding_model: bge-m3           # model requested from that backend (required)
    threshold: 0.92                   # cosine similarity; absent = hub semantic_similarity_bps
    max_entries: 1000                 # vectors held in the per-router index, 1..=1000000
    ttl: "5m"                         # entry TTL; absent = response_cache.ttl
    embedding_timeout_ms: 2000        # embedding call bound, 100..=60000; expiry is a miss
    max_input_bytes: 8192             # longer final user messages skip the path, 1..=1048576

Tier 3: Shared External Cache (Redis backend)¶

response_cache:
  enabled: true
  backend: redis

  redis:
    # Redis/Valkey connection URL
    url: "redis://redis:6379"

    # rediss:// for TLS; or set tls: true with redis:// URL
    # url: "rediss://redis:6380"

    # Connection pool size (default: 8)
    pool_size: 8

    # Key namespace prefix (must not contain glob characters)
    key_prefix: "cr:resp:"

    # Connection timeout in milliseconds (default: 3000)
    connect_timeout_ms: 3000

    # Per-command timeout in milliseconds (default: 1000)
    command_timeout_ms: 1000

    # Fallback capacity for in-memory cache when Redis is unreachable (default: 1000)
    fallback_capacity: 1000

    # TTL for fallback in-memory entries in seconds (default: 300)
    fallback_ttl_seconds: 300

    # Fall back to in-memory cache on Redis failure (default: true)
    fallback_to_memory: true

Tier 4: KV Cache Index¶

kv_cache_index:
  enabled: true

  # Index backend: "memory" (default) or "redis"
  # When "redis", reuses the connection pool from response_cache.redis
  backend: memory

  # Maximum prefix hash entries tracked (default: 100000)
  # Range: 100 to 10,000,000
  max_entries: 100000

  # TTL for index entries in seconds (default: 600)
  # Range: 1 to 86400
  entry_ttl_seconds: 600

  # Backend scoring weights (overlap + load + health must sum to 1.0)
  scoring:
    overlap_weight: 0.6
    load_weight: 0.3
    health_weight: 0.1

    # Minimum best-overlap to activate KV-aware routing (default: 0.3)
    # If no backend exceeds this, falls back to configured strategy
    min_overlap_threshold: 0.3

    # Tier weight multipliers (independent of the three main weights)
    # Applied to the raw overlap score before the weighted sum
    gpu_tier_weight: 1.0       # Multiplier for GpuHot (GPU-resident) data
    storage_tier_weight: 0.6   # Multiplier for StorageWarm (offloaded) data

  # Tiered storage awareness (GPU hot vs. external storage warm)
  storage_offloading:
    enabled: false               # Enable storage tier tracking (default: false)
    treat_eviction_as_offload: true  # Treat cache_evicted as offload to warm (default: true)

  # continuum-kv-listener SSE streams to subscribe to for KV cache events
  event_sources:
    - backend_name: vllm-1
      endpoint: "http://kv-listener.internal:7817/events/vllm-1"
      reconnect_interval_ms: 5000

    - backend_name: vllm-2
      endpoint: "http://kv-listener.internal:7817/events/vllm-2"
      reconnect_interval_ms: 5000

Supported endpoint schemes for event_sources[].endpoint: http, https, ws, wss.

Metrics¶

All KV cache metrics use the prefix continuum_. Label values are sanitized against allowlists to prevent cardinality explosion.

Tier 1: Prefix Routing¶

continuum_prefix_routing_requests_total is recorded once per selection and only when the PrefixAwareHash strategy arm ran, so under any other selection_strategy it stays at zero by design. See Routing decision labels for what each label means in terms of the ring walk.

Example PromQL:

# Overflow rate (CHWBL load cap activations)
rate(continuum_prefix_routing_requests_total{strategy="overflow"}[5m])
/ rate(continuum_prefix_routing_requests_total[5m])

# Per-backend load distribution
continuum_prefix_routing_backend_distribution

Tier 2: Response Cache¶

Example PromQL:

# Cache hit rate over 5 minutes
rate(continuum_response_cache_requests_total{result="hit"}[5m])
/ rate(continuum_response_cache_requests_total{result=~"hit|miss"}[5m])

# Cache bypass rate (non-deterministic requests)
rate(continuum_response_cache_requests_total{result="skip"}[5m])

Tier 3: Redis Backend¶

Example PromQL:

# Redis P99 GET latency
histogram_quantile(0.99,
  rate(continuum_cache_redis_latency_seconds_bucket{operation="get"}[5m])
)

# Redis error rate
rate(continuum_cache_redis_errors_total[5m])

# Alert: fallback active
continuum_cache_fallback_active == 1

Tier 4: KV Cache Index¶

The two routing labels are recorded at the same seam, once per selection: kv_aware when the scorer won and its backend was forwarded to, and fallback when the scorer ran with a prefix key but no backend cleared the combined-score threshold, so the configured strategy decided instead. A selection the scorer never attempted (no scorer registered, or no prefix key) records neither. continuum_kv_index_overlap_score observes the winning combined score of a kv_aware decision, which is the full weighted sum of overlap, load, and health, not the overlap term alone.

Example PromQL:

# KV-aware routing activation rate
rate(continuum_kv_index_routing_decisions_total{decision="kv_aware"}[5m])
/ rate(continuum_kv_index_routing_decisions_total[5m])

# Event drop rate per backend (indicates backpressure)
rate(continuum_kv_event_dropped_total[5m])

# P50 overlap score for routed requests
histogram_quantile(0.50, rate(continuum_kv_index_overlap_score_bucket[5m]))

# Disconnected event consumers
continuum_kv_consumer_connected == 0

Admin Endpoints¶

All admin endpoints are under the /admin prefix and require authentication if configured.

Prefix Routing¶

GET /admin/prefix-routing/stats¶

Returns prefix routing statistics including routing decision counts, overflow rate, backend load distribution, and CHWBL configuration.

Example response:

{
  "enabled": true,
  "config": {
    "max_prefix_length": 1024,
    "load_factor_epsilon": 0.25,
    "virtual_nodes": 150,
    "anthropic_cache_control_injection": false
  },
  "routing_decisions": {
    "total": 4926,
    "prefix_hash": 4821,
    "overflow": 93,
    "fallback": 12,
    "overflow_rate": "0.0189"
  },
  "backend_distribution": [
    { "backend": "vllm-1", "in_flight_requests": 4 },
    { "backend": "vllm-2", "in_flight_requests": 3 }
  ],
  "unique_prefixes": 247
}

Response Cache¶

GET /admin/response-cache/stats¶

Returns response cache statistics: hit/miss/skip counts, hit rate, entry count, memory usage, and Redis connection info (when applicable).

Example response:

{
  "enabled": true,
  "backend_type": "redis",
  "entries": 1243,
  "capacity": 5000,
  "requests": {
    "hit": 8912,
    "miss": 2341,
    "skip": 441,
    "total": 11694
  },
  "hit_rate": "0.7924",
  "evictions": 0,
  "size_bytes": 0,
  "config": {
    "backend": "redis",
    "ttl": "30m",
    "capacity": 5000,
    "max_response_size": 1048576,
    "max_stream_buffer_size": 10485760
  },
  "redis": {
    "connections": {
      "active": 3,
      "idle": 5
    },
    "fallback_active": false,
    "errors": {
      "connection": 0,
      "timeout": 2,
      "other": 0
    }
  }
}

POST /admin/response-cache/invalidate¶

Invalidates cached responses. Accepts a JSON body:

{ "clear_all": true }

Example response:

{
  "success": true,
  "action": "clear_all",
  "cleared_entries": 1243
}

KV Cache Index¶

GET /admin/kv-index/stats¶

Returns KV cache index statistics: entry counts, routing decision breakdown, query latency counters, and overlap score counts.

Example response:

{
  "enabled": true,
  "config": {
    "backend": "memory",
    "max_entries": 100000,
    "entry_ttl_seconds": 600,
    "event_sources_count": 2,
    "scoring": {
      "overlap_weight": 0.6,
      "load_weight": 0.3,
      "health_weight": 0.1,
      "min_overlap_threshold": 0.3
    }
  },
  "index": {
    "prefix_count": 312,
    "entry_count": 618,
    "total_hits": 12490,
    "total_evictions": 83
  },
  "event_sources": [
    {
      "backend_name": "vllm-1",
      "connected": true,
      "events_received": 8412,
      "events_dropped": 0,
      "last_event_at": "2026-03-13T10:24:17Z",
      "reconnect_count": 0
    }
  ],
  "routing_decisions": {
    "kv_aware": 9841,
    "fallback": 2649,
    "total": 12490
  },
  "query_latency_count": 12490,
  "overlap_score_count": 9841
}

GET /admin/kv-index/backends¶

Returns per-backend KV cache event statistics: events received, processed, dropped, connection status, and index event counts (created/evicted).

Example response:

{
  "enabled": true,
  "backends": [
    {
      "backend_name": "vllm-1",
      "connection": {
        "connected": true,
        "reconnect_count": 0,
        "last_event_at": "2026-03-13T10:24:17Z"
      },
      "events": {
        "received": 8412,
        "dropped": 0,
        "index_created": 7981,
        "index_evicted": 431
      }
    }
  ]
}

POST /admin/kv-index/clear¶

Clears all entries from the KV cache index. The index rebuilds automatically from incoming events. Intended for debugging.

Example response:

{
  "success": true,
  "entries_before_clear": 618,
  "cleared_entries": 2
}

Deployment Guide¶

Tier 1 Only (Minimal Configuration)¶

Enable prefix routing for GPU KV cache locality without any external dependencies:

prefix_routing:
  enabled: true
  load_factor_epsilon: 0.25
  virtual_nodes: 150

Effective when multiple instances of the same model run across backends and system prompts are long (>128 tokens).

Tier 1 + 2 (Response Cache)¶

Add response caching to eliminate repeated deterministic requests entirely:

prefix_routing:
  enabled: true

response_cache:
  enabled: true
  backend: memory
  capacity: 5000
  ttl: "10m"

Effective when the application makes repeated identical requests (e.g., document QA with fixed system prompts and fixed queries).

Tier 1 + 2 + 3 (Distributed Response Cache)¶

For multi-instance deployments, share the response cache across all router instances:

prefix_routing:
  enabled: true

response_cache:
  enabled: true
  backend: redis
  ttl: "30m"
  redis:
    url: "redis://redis-service:6379"
    pool_size: 16
    key_prefix: "cr:resp:"
    fallback_to_memory: true
    fallback_capacity: 2000

Redis/Valkey must be accessible from all router pods. Use rediss:// URL or tls: true for encrypted connections.

All Tiers (Full KV-Aware Routing)¶

Enable all tiers for maximum GPU cache reuse:

prefix_routing:
  enabled: true
  load_factor_epsilon: 0.20

response_cache:
  enabled: true
  backend: redis
  ttl: "30m"
  redis:
    url: "redis://redis-service:6379"
    pool_size: 16

kv_cache_index:
  enabled: true
  backend: redis   # shares the pool from response_cache.redis
  max_entries: 500000
  entry_ttl_seconds: 900
  scoring:
    overlap_weight: 0.6
    load_weight: 0.3
    health_weight: 0.1
    min_overlap_threshold: 0.25
    gpu_tier_weight: 1.0       # GPU-resident data gets full overlap credit
    storage_tier_weight: 0.6   # Offloaded data is still valuable but discounted
  storage_offloading:
    enabled: true              # Track GPU hot vs. storage warm tiers
    treat_eviction_as_offload: true
  event_sources:
    - backend_name: vllm-1
      endpoint: "http://kv-listener.internal:7817/events/vllm-1"
    - backend_name: vllm-2
      endpoint: "http://kv-listener.internal:7817/events/vllm-2"

Engine Event Source Requirements¶

Start continuum-kv-listener on the internal network and configure each engine to publish KV events to its ZMQ endpoint. The router consumes only the listener's SSE endpoint through kv_cache_index.event_sources[]; the engine container does not need an extra HTTP producer port.

continuum-kv-listener --config config.kv-listener.yaml

For vLLM, set kv_events_config or --kv-events-config to the listener endpoint used for the matching backend, for example {"publisher":"zmq","endpoint":"tcp://kv-listener.internal:5557","topic":""}. SGLang uses the same ZMQ event classes and can publish to a sibling source entry. Keep the listener bind address and network policy internal because token-derived event data crosses this port before detokenization.

For native TensorRT-LLM, set the listener source to engine: trtllm and point trtllm_endpoint at the exact trtllm-serve /kv_cache_events URL. Enable block reuse and configure a positive event buffer size in TensorRT-LLM. The poll interval, request timeout, and response byte ceiling are bounded independently. The URL host must appear in allowed_publishers, redirects are rejected, and the host should resolve only inside the deployment network.

trtllm-serve does not expose a public detokenize endpoint, so a native TensorRT-LLM source depends on the router-issued cache salt integration. The first usable stored block must carry a salt matching the source backend; later blocks may inherit that resolved chain identity. The listener neither logs nor publishes salts, raw response bodies, or token IDs. If the salt is missing or cannot be verified, the chain is discarded rather than attributed to a prefix.

Redis/Valkey Sizing¶

For response cache sizing, estimate:

• Average serialized response size: 2–10 KB per entry
• capacity = (target_hit_rate * rps * avg_unique_rate) / eviction_frequency

For the KV index:

• Each (prefix, backend) entry consumes approximately 200 bytes in memory
• Set max_entries to at least num_unique_prefixes * num_backends * 2 for headroom

High Availability Considerations¶

• The response cache and KV index tolerate Redis failure via automatic in-memory fallback (Tier 3).
• The KV index rebuilds from the SSE streams on restart; no persistence is required.
• Prefix routing (Tier 1) has no external dependencies and is always available.
• Deploy Redis with replication (Sentinel or Cluster) if cache persistence across Redis restarts is required.

Performance Characteristics¶

Tier 1: Prefix Routing¶

• Prefix key extraction (SHA256): < 10 µs per request
• CHWBL ring lookup: O(log N) where N = virtual_nodes * num_backends; < 5 µs for typical deployments
• No network I/O; operates entirely in-process

Tier 2: Response Cache¶

• In-memory cache lookup: < 1 µs
• Cache key computation (SHA256): < 5 µs
• Cache hit serves the entire response without any backend latency
• Semantic lookup: one embedding round trip to the pinned backend plus a partition-local scan of at most max_entries normalized vectors

Tier 3: Redis Backend¶

• Redis GET latency (LAN): 0.1–2 ms typical; P99 < 5 ms
• Redis SET latency: similar to GET
• Command timeout default: 1 second; operations exceeding this activate fallback

Tier 4: KV Cache Index¶

• InMemoryKvIndex.query_backends(): < 100 µs (DashMap read, no allocation on empty result)
• RedisKvIndex.query_backends(): same as Redis GET latency (0.1–2 ms)
• KvOverlapScorer.prepare(): one query_backends() call per unique prefix per 100 ms window
• KvOverlapScorer.score(): < 1 µs (synchronous read from pre-fetched cache)
• 1000 scoring calls: < 100 ms total (verified by unit benchmark in scorer.rs)

Expected Gains¶

The following are illustrative estimates based on typical LLM workload patterns:

Actual gains depend on workload prefix overlap, GPU memory capacity, and backend configuration.