Health, Caching & Logging¶

Health Checks Section¶

Configures backend health monitoring:

health_checks:
  enabled: true                      # Set false to skip every backend probe
  interval: "30s"                  # Check frequency
  timeout: "10s"                   # Request timeout
  unhealthy_threshold: 3           # Failures before marking unhealthy
  healthy_threshold: 2             # Successes before marking healthy
  endpoint: "/health"              # Endpoint to check (per-type defaults below)
  warmup_check_interval: "1s"      # Accelerated interval during warmup
  max_warmup_duration: "300s"      # Maximum warmup detection duration
  block_startup: true              # Gate listener binding on pre-warm and the initial health check
  prewarm_timeout: "10s"           # Per-request bound on the startup pre-warm probes

The top-level health_checks.interval and health_checks.timeout values are authoritative at startup and after reload. The legacy timeouts.health_check block remains parseable for migration compatibility but is deprecated and ignored; when explicitly specified, conflicting effective values emit a warning. Omitting timeouts.health_check emits no warning.

Health Check Process:

1. Router queries the health endpoint on each backend
2. Successful responses increment success counter
3. Failed responses increment failure counter
4. Backends marked unhealthy after reaching failure threshold
5. Backends marked healthy after reaching success threshold
6. Only healthy backends receive traffic

Accelerated Warmup Health Checks¶

The router supports accelerated health checks during backend warmup, which is particularly useful for backends like llama.cpp that return HTTP 503 while loading models.

Backend States:

Warmup Configuration:

How it works:

1. When a backend returns HTTP 503, it enters the warming_up state
2. Health checks switch to the accelerated interval (default: 1 second)
3. Once the backend returns HTTP 200, it becomes ready and returns to normal interval
4. If warmup exceeds max_warmup_duration, the backend is marked as unhealthy

This reduces model availability detection latency from up to 30 seconds (worst case) to approximately 1 second.

Startup Behavior¶

health_checks.block_startup (default: true) controls whether the router binds its listeners before or after the first backend health check completes.

• Liveness vs. readiness: /health reports router liveness only, never backend health. /health/ready reports serving-pool readiness: it returns 503 while no enabled, non-internal backend is active in the pool and 200 once at least one such backend is available (draining backends are already absent from the active pool). This readiness boundary excludes upstream health and circuit-breaker state, which continue to govern request selection. Set block_startup: false when an external readiness probe enforces a short deadline, so a single unreachable or stalling backend cannot push the pre-bind window past that deadline.
• Probe and pre-warm timeouts: Every health-check request is bounded by timeout regardless of block_startup. Startup pre-warm requests use health_checks.prewarm_timeout (default 10s), preventing a stalling backend from holding the router's shared client timeout (the 600s streaming total by default) during startup. Both the Anthropic and generic pre-warm paths read prewarm_timeout; because pre-warm runs only at startup, changing it takes effect on the next restart.
• Dedicated probe connection pool: Health checks and model catalog fetches run on a dedicated HTTP client that shares the proxy client's TLS, proxy, timeout, keep-alive, and pool settings but uses a separate connection pool. It negotiates and decodes gzip for Router-owned probe and discovery responses. The inference proxy and SSE clients continue to request identity bodies. Proxied streams that saturate a backend connection's MAX_CONCURRENT_STREAMS cannot starve a health probe or cause it to time out. Pre-warm still warms the proxy connection pool.

health_checks:
  block_startup: false             # Bind listeners first; converge health in the background

Per-Backend Health Check Configuration¶

Each backend type has sensible default health check endpoints. You can override these defaults with a custom health_check configuration per backend.

Default Health Check Endpoints by Backend Type:

Fallback Behavior:

Built-in strategies try a fallback only when the primary returns HTTP 404 or 405, which means the path or method is unsupported. Authentication failures, permission failures, rate limits, timeouts, connection failures, 5xx outages, and 503 warmup never fall through to a public catalog. This keeps those outcomes visible instead of masking them as healthy.

Catalog-sized built-in fallbacks are single-flighted per backend and effective URL, authentication, organization, and extension-header context. The exact result, including a transport or HTTP failure, is retained for five minutes. Repeated health ticks still retry the lightweight primary endpoint, but issue at most one catalog fallback request in that window; a configuration or credential-context change invalidates the prior verdict. Reused verdicts do not advance health thresholds, record a new latency sample, or emit recovery events. Startup is the bounded exception: its finite initial rounds use fresh observations so healthy_threshold can be satisfied before listener binding. During ordinary monitoring, if no lightweight endpoint works, each fresh fallback observation arrives at most once per five minutes. Reaching a healthy or unhealthy threshold of N can therefore take approximately N × 5 minutes, plus probe scheduling and request timeouts. When a public or unknown catalog returns 200, it establishes HTTP liveness and catalog compatibility only. Candidate-probe credential status remains unknown unless an authenticated health resource or documented authenticated first-party catalog has established valid. The built-in OpenRouter /api/v1/key probe establishes this evidence; a later public catalog response preserves it.

These bounds apply to built-in defaults. An explicit per-backend health_check is sent with its configured endpoint, method, body, accepted statuses, timeout, provider-appropriate credential header, and merged request-extension headers. Static native Gemini API keys use x-goog-api-key; explicit Basic credentials and other static credentials keep their configured Authorization scheme. A configured catalog endpoint therefore runs at the configured health interval by operator choice. An explicitly accepted 404 or 405 succeeds before fallback selection; otherwise those two statuses try the configured fallback list.

Aliases sharing a URL keep separate health state, credentials, headers, and endpoint configuration by backend name. Reordering preserves their histories; an unambiguous rename preserves accumulated health, while results dispatched before a reload cannot update the new configuration. A cached warmup result still enforces max_warmup_duration on schedule without counting another failure sample.

Unix-socket backends use the same strategies, catalog cadence, authentication, and explicit overrides. Requests use an origin-form path on the configured socket, including native Gemini model paths. Health checks read only response headers, capped at 64 KiB, and apply the configured timeout to the entire connect/write/read operation; they do not buffer catalog bodies.

Custom Health Check Configuration:

backends:
  - name: vllm-custom
    type: vllm
    url: http://localhost:8000
    models:
      - my-model
    health_check:
      endpoint: /custom-health          # Primary endpoint
      fallback_endpoints:               # Tried if primary returns 404 or 405
        - /health
        - /v1/models
      method: GET                       # HTTP method: GET, POST, or HEAD
      timeout: 10s                      # Override global health check timeout
      accept_status:                    # Status codes indicating healthy
        - 200
        - 204
      warmup_status:                    # Status codes indicating model loading
        - 503

Health Check Configuration Options:

Example: Anthropic-Style Health Check:

For a custom backend that deliberately treats error statuses as liveness evidence, list them explicitly:

backends:
  - name: custom-api
    type: generic
    url: http://localhost:9000
    models:
      - custom-model
    health_check:
      endpoint: /api/v1/health
      method: POST
      body:
        check: true
      accept_status:
        - 200
        - 400    # Bad request means server is up

Timeouts Section¶

Request handling timeouts live under timeouts:, with separate budgets for non-streaming, streaming, and image-generation requests, plus model-specific overrides:

timeouts:
  connection: "10s"                # TCP connection establishment timeout
  request:
    standard:
      first_byte: "30s"            # Deprecated and inert; `total` is the real budget
      total: "180s"                # Total request timeout (default cap 540s)
    streaming:
      first_byte: "120s"           # Enforced deadline for the first SSE chunk
      chunk_interval: "30s"        # Max time between later chunks
      total: "600s"                # Total streaming timeout (default cap 1200s)
    model_overrides:
      gpt-5-latest:
        streaming:
          total: "1200s"           # Per-model override
  limits:                          # Optional; pinned at startup, restart to change
    max_standard_timeout: "900s"   # Default 540s, absolute ceiling 1080s

Which field applies where¶

first_byte does not mean the same thing on the two paths, and only one of the two does anything.

• streaming.first_byte is enforced. A streaming response emits its first SSE chunk long before the stream ends, so a first-chunk deadline is a real signal separate from the total budget. It defaults to 120s, because reasoning models routinely exceed 60s to their first token, and chunk_interval takes over from the first chunk onward.
• standard.first_byte and image_generation.first_byte are deprecated and inert, and will be removed in the next major release. A non-streaming completion is a buffered body whose headers arrive only once the whole completion is generated, so time to first byte is the full generation time. total is the only budget on those paths. Both fields are still parsed and still bounded by first_byte <= total, and a non-default value produces a load-time deprecation warning.

So a reasoning model that keeps timing out needs a larger streaming.first_byte on the streaming path and a larger standard.total on the non-streaming path. Raising standard.first_byte changes nothing.

Validation ceilings¶

Every cap on this section is itself configurable through timeouts.limits, which is resolved once from the boot configuration and pinned for the process lifetime. Hot reload, the Admin API, and control-plane config sync all reject a candidate that changes it, so a runtime path cannot widen the router's own DoS guard; changing the ceilings means editing the file and restarting. Each field also has a compiled-in absolute ceiling, and a value above it is a load error rather than a silent clamp.

Timeout Considerations:

• Long totals accommodate slow model inference
• Streaming requests may take longer than non-streaming
• Balance between user experience and resource usage
• Raising max_standard_timeout costs nothing until an operator raises standard.total itself; at that point a hung upstream holds a request slot for the larger budget, so pair it with connection, health checks plus the circuit breaker, and server.max_concurrent_requests

Retry Section¶

Global retry configuration for resilience:

retry:
  max_attempts: 3                  # Maximum retry attempts
  initial_delay: "100ms"           # Initial delay between retries
  max_delay: "30s"                 # Cap on retry delays
  backoff_multiplier: 2.0          # Exponential backoff multiplier
  jitter: true                     # Add random jitter
  retryable_status_codes: [429, 502, 503, 504]
  prefer_untried_backends: false   # Retry onto a backend this request has not dialed yet

Retry Strategy:

• Exponential backoff: delays increase exponentially (100ms, 200ms, 400ms...)
• Jitter: adds randomness to prevent thundering herd
• Max delay: prevents extremely long waits

A retry may re-select the backend that just failed: selection does not exclude earlier attempts, and in a weighted pool the heavier backend keeps receiving the retry.

Set prefer_untried_backends: true to have a retry select among the backends this request has not dialed yet. When every eligible backend has been dialed, selection falls back to the full eligible set, so a single-backend deployment still gets every attempt. This is a bias, not a replay guarantee.

A backend whose requests have side effects sets replay: disabled, which ends the request at the first failure that may have reached it, with no retry and no fallback; see Replay Policy.

Model Aggregation Section¶

The model_aggregation section configures the aggregated model-list cache, the per-backend model fetcher, deduplication, the force-refresh endpoints, and the background refresh task. Every field has a default, so the section is optional; a resolved snapshot is applied at startup. Changing it requires a restart.

model_aggregation:
  cache_ttl: "60s"                  # Hard TTL for an aggregated model list
  soft_ttl_ratio: 0.8               # Stale-but-usable fraction of cache_ttl, (0.0, 1.0]
  request_timeout: "5s"             # Per-backend model-list fetch timeout
  max_retries: 2                    # Fetch retries per backend
  retry_delay: "500ms"              # Delay between fetch retries
  max_models_per_backend: 1000      # Cap on models from any single backend (> 0)
  max_total_models: 5000            # Cap on total models across backends (> 0)
  max_response_size: 10485760       # Max backend response size in bytes (> 0)
  empty_response_base_ttl: "5s"     # Empty-response backoff base
  empty_response_max_ttl: "60s"     # Empty-response backoff cap (>= base)
  max_cache_entries: 100            # Max cache entries, LRU beyond this (> 0)
  deduplication_strategy: merge_backends  # first_wins | last_wins | merge_backends
  allow_force_refresh: true         # false => POST /v1/models/refresh returns 403
  background_refresh:
    enabled: true                   # Proactive pre-expiry cache refresh
    check_interval: "10s"           # Background refresh polling cadence

Fetch budget for a backend that stopped answering¶

Because model refreshes fan out to all backends concurrently, a refresh takes as long as its slowest backend. At the defaults, an unresponsive backend costs (max_retries + 1) × request_timeout + max_retries × retry_delay (16 seconds) on every refresh.

To bound that tail latency, the router adjusts the fetch budget based on recent health:

• Single-attempt probe after failure: If a backend's previous refresh failed with a transient error (timeout, connection failure, or 5xx), the next refresh probes it with a single attempt at request_timeout and no retries (5 seconds at the defaults instead of 16). The per-attempt timeout itself is not shortened, so a slow-but-working backend can still respond and regain its full retry budget on its first success.
• Full budget on first refresh and fleet-wide failures: Backends with no failure history (on initial process startup or when newly added) always receive the full retry budget. If every backend fails during the same refresh, any backend that was probed only once is given its remaining retries before the result is published, preventing a brief fleet-wide blip from caching an empty model list.
• Metrics and health impact: A shortened probe that fails is recorded as an ordinary transient error in model_transient_errors_total. The backend's last-known models remain in the published list, and its health status is unchanged. This behavior is automatic after a single failed refresh.

Cache Stampede Prevention¶

The router implements three strategies to prevent cache stampede (thundering herd problem):

1. Singleflight Pattern: Only one equivalent aggregation request runs at a time
2. Stale-While-Revalidate: Stale data remains usable while refresh runs (past the soft_ttl_ratio point of cache_ttl)
3. Background Refresh: When background_refresh.enabled, a check_interval loop refreshes entries after the soft TTL

Discovery fan-out remains tied to cache semantics: the first empty-cache/startup pass, backend or configuration invalidation, soft/hard TTL revalidation, an unhealthy-to-healthy transition, and an explicit refresh can request catalogs. Healthy-transition events are debounced for five seconds, and every trigger shares the same revalidation singleflight. A periodic health success that does not change state emits no event and cannot create another catalog fan-out.

Stale serving also extends past hard expiry (cache_ttl). When cache_ttl expires, the router still returns the previously aggregated list immediately while triggering a single background re-aggregation, avoiding a synchronous fan-out across all backends. This stale window is capped at three times cache_ttl (180 seconds at the default 60s TTL); beyond that limit, requests block until fresh data arrives.

The same retention applies on cache invalidation. When a hot reload modifies the backend set, the cached list is marked expired rather than dropped, and its age timer resets from the invalidation timestamp. Clients that list models immediately after a backend update receive the retained list while the new topology aggregates in the background. POST /v1/models/refresh bypasses retention, clearing the cache and blocking until fresh data is fetched.

Empty results use bounded exponential backoff between empty_response_base_ttl and empty_response_max_ttl. Use the rate-limited POST /v1/models/refresh endpoint for an immediate refresh; when allow_force_refresh: false, that endpoint and the Admin refresh endpoint return 403.

Response Cache Section¶

Caches complete LLM responses for deterministic (temperature == 0) requests, both non-streaming and streaming. Repeated identical requests are served from memory without calling the backend.

response_cache:
  enabled: true                     # Enable response caching (default: false)
  backend: memory                   # Cache backend: "memory" (default), "redis", or "tiered"
  capacity: 1000                    # Maximum number of cached responses (LRU eviction)
  ttl: "5m"                         # Time-to-live for cached entries (e.g., "5m", "1h")
  max_response_size: 1048576        # Maximum response body size in bytes (default: 1 MiB)
  max_stream_buffer_size: 10485760  # Maximum streaming buffer size in bytes (default: 10 MiB)

Cache Eligibility¶

A response is cached only when all of the following conditions hold:

• response_cache.enabled is true
• The request's temperature field is 0 or absent (deterministic output)
• The response body does not exceed max_response_size
• For non-streaming requests: the response does not contain finish_reason: "error" in any choice
• For streaming requests: the stream completes successfully (receives the final [DONE] event); interrupted or errored streams are discarded and not cached

Non-eligible requests are forwarded to the backend normally.

Guardrail Policy Scope¶

When guardrails are configured, every response-cache namespace (exact, prompt-prefix, and semantic) includes a scope derived from the live guardrail policy, the router version, the matched guardrails.routes entry, and whether the caller holds a bypass_api_keys key. Because a cache hit skips output-stage guardrails, cached entries are served only to requests that match the policy scope under which they were stored:

• Policy changes and upgrades: After a guardrail policy change (via hot reload, Admin API write, or hub policy delivery) or a router upgrade, each distinct prompt misses once and is re-cached under the new scope. Entries from the previous scope expire through normal TTL or eviction.
• In-flight and fail-open exclusions: Responses are not cached if the policy changed while the request was in flight, or if a guardrail provider failed under on_error: fail_open.
• Unrelated changes: Changes outside guardrails (such as deploying a new classifier model behind the same endpoint) do not alter the cache scope. Deployments without guardrails keep their cache keys unchanged. See Response cache and policy changes.

Streaming Cache Behaviour¶

When caching is enabled and temperature is 0, streaming (stream: true) chat completions are also eligible for caching:

1. SSE events are buffered in memory alongside normal stream forwarding to the client; there is no observable latency increase.
2. On successful stream completion (final [DONE] event), the router reconstructs a complete chat.completion JSON object from the buffered chunks and stores it in the cache.
3. If the stream is interrupted (client disconnect, timeout, backend error), the buffer is discarded and nothing is stored.
4. On cache hit, the cached response is replayed to the client as a synthetic SSE stream (text/event-stream) with an X-Cache: HIT header.
5. The max_stream_buffer_size setting limits the streaming buffer. Streams whose total buffered content exceeds this threshold are not cached.

X-Cache Response Header¶

Every chat completion response includes an X-Cache header:

Cache Configuration Options¶

Redis/Valkey Backend¶

By default the response cache stores entries in process memory. To share the cache across multiple router instances or survive restarts, configure a Redis or Valkey backend.

Build requirement: none for official artifacts. Release binaries and the published container images compile the redis-cache Cargo feature, so this section only needs configuration. A source build gets it from the default full set only if you add the feature:

cargo build --release --features redis-cache

Supported topology: a single managed endpoint. This includes a standalone server, a provider-managed HA endpoint that fails over behind a stable DNS name or virtual IP (ElastiCache with automatic failover, Azure Cache for Redis, Memorystore, or managed Valkey), or a proxy presenting a cluster as a single address. Direct Redis Sentinel and Redis Cluster connections are not supported: the router uses a single-endpoint client without Sentinel master discovery or cluster slot routing. URLs using redis+sentinel://, redis+cluster://, or comma-separated multi-host lists are rejected at configuration load time. See Deployment for deployment details.

Configuration:

response_cache:
  enabled: true
  backend: redis                     # Select Redis backend
  ttl: "5m"
  redis:
    url: "redis://localhost:6379"    # plain TCP
    # url: "rediss://redis.example.com:6380"   # TLS
    # url: "redis://:password@localhost:6379"  # with auth
    pool_size: 8                     # connection pool size (default: 8)
    key_prefix: "cr:resp:"           # key namespace prefix (default: "cr:resp:")
    connect_timeout_ms: 3000         # connection timeout (default: 3000)
    command_timeout_ms: 1000         # per-command timeout (default: 1000)
    tls: false                       # use TLS (default: false)
    fallback_to_memory: true         # fallback to in-memory on failure (default: true)

Credentials: pass credentials in the URL userinfo via an environment variable reference, such as url: "${REDIS_URL}" with REDIS_URL=rediss://:$PASSWORD@cache.example.com:6380. Deployment charts inject this variable from an existing Secret. continuum-router config validate warns when credentials are hardcoded in the configuration file or sent over a plaintext redis:// connection. Router logs always mask the password.

Outage behaviour:

• Response cache (fallback_to_memory): With fallback_to_memory: true (the default), an unreachable Redis instance causes the router to log a warning and serve from a per-replica in-memory cache (fail-open). A background health monitor sends PING periodically and switches back once Redis recovers. Set fallback_to_memory: false for fail-closed behavior: the cache reports the outage as a logged miss rather than serving from local memory, so requests still reach the backend without caching until Redis returns.
• Distributed rate limiting (rate_limiting.storage: redis): Fail-open by default (redis.fallback_to_memory: true). During a Redis outage, each replica falls back to its own in-memory token bucket, multiplying the effective fleet rate limit by the replica count; size your limits with that degraded ceiling in mind. Every fallback is counted by rate_limit_redis_unavailable_total{dimension, action="fallback"} (a per-replica counter; the fleet signal is the sum across replicas). Set redis.fallback_to_memory: false for fail-closed behavior: the check refuses with a 503 (type: service_unavailable, Retry-After: 5) until Redis answers, counted as action="refused", keeping the shared quota exact at the cost of availability during the outage. Under fail-closed, a per_backend charge that cannot reach Redis refuses that dispatch like an exhausted backend. A check that gets no answer within 500 ms counts as unreachable, so a partitioned Redis does not stall requests. The Redis backend is built at startup, so changing fallback_to_memory requires a restart.

Key safety: clear() uses SCAN + DEL with the configured prefix pattern instead of FLUSHDB. Only keys matching the prefix are removed, making it safe to share a Redis instance with other applications.

S3-Compatible Tiered Backend¶

The tiered backend composes a fast, bounded L1 cache with a virtually unlimited L2 cache backed by an S3-compatible API. Large responses are stored only in L2 to avoid eviction pressure on L1. When an L2 hit occurs, the value is optionally promoted back to L1 for faster subsequent access.

Any S3-compatible storage can be used as L2, including AWS S3, MinIO, Ceph, or VAST Data.

Build requirement: The binary must be compiled with the s3-cache Cargo feature:

cargo build --release --features s3-cache

Tiered cache write path:

1. Every set() writes to L2 (S3) unconditionally.
2. If the value is smaller than l1.max_value_size, it is also written to L1.
3. Values exceeding the threshold are demoted to L2-only; the demotion counter is incremented.

Tiered cache read path:

1. L1 is checked first. On hit, the value is returned immediately.
2. On L1 miss, L2 (S3) is queried.
3. On L2 hit, if tiered.promote_on_hit is true and the value fits in L1, it is promoted back to L1 with tiered.l1_promotion_ttl as the TTL.

TTL enforcement: Each S3 object carries an expires-at metadata field (Unix timestamp). On GET, if the timestamp has passed, the object is deleted lazily and a cache miss is returned. Use l2.ttl_override to keep L2 entries alive longer than the global ttl.

Configuration:

response_cache:
  enabled: true
  backend: tiered                    # Select tiered L1/L2 backend
  ttl: "5m"                          # Default TTL for both tiers

  l1:                                # L1 (hot) tier: fast, bounded
    type: memory                     # "memory" or "redis" (redis requires redis-cache feature)
    max_value_size: 1048576          # Values larger than this go to L2 only (default: 1 MiB)

  l2:                                # L2 (warm) tier: S3-compatible storage
    type: s3
    endpoint: "https://s3.example.com:8080"
    bucket: "llm-response-cache"
    key_prefix: "response-cache/"    # Object key prefix (default: "response-cache/")
    region: "us-east-1"              # Region string (default: "us-east-1")
    access_key: "${S3_ACCESS_KEY}"   # Supports env var expansion
    secret_key: "${S3_SECRET_KEY}"
    ttl_override: "24h"              # Optional: keep L2 entries longer than global TTL

  tiered:                            # Promotion/demotion behavior
    promote_on_hit: true             # Promote L2 hits back to L1 (default: true)
    l1_promotion_ttl: "5m"          # TTL for promoted entries (default: "5m")

Metrics: The s3-cache build adds Prometheus counters for L1/L2 hit rates, promotion/demotion counts, and an S3 operation latency histogram (continuum_cache_s3_latency_seconds).

Using Redis as L1: Combine the redis-cache and s3-cache features to use Redis as the L1 tier:

cargo build --release --features redis-cache,s3-cache

response_cache:
  backend: tiered
  l1:
    type: redis                      # Use Redis as L1 for cross-instance sharing
    max_value_size: 1048576
  redis:                             # Shared Redis config reused by l1
    url: "redis://localhost:6379"
    pool_size: 8
  l2:
    type: s3
    endpoint: "https://s3.example.com:8080"
    bucket: "llm-response-cache"
    access_key: "${S3_ACCESS_KEY}"
    secret_key: "${S3_SECRET_KEY}"

Limitations¶

• Streaming cache is supported for OpenAI-compatible backends only. Anthropic and Gemini native streaming formats are not cached.
• The Anthropic (/anthropic/v1/messages) and OpenAI (/v1/chat/completions) endpoints each maintain the same shared cache instance when response_cache.enabled: true.
• Without the Redis backend configured, cache is stored in-memory and does not survive server restarts.

KV Cache Index Section¶

Enables real-time KV cache state tracking for precise overlap-scored routing. vLLM and SGLang publish KV events over ZMQ to continuum-kv-listener; the listener detokenizes the block chains, computes router-compatible prefix hashes, and exposes per-backend SSE streams. The router subscribes to those listener streams, maintains an index of cache state across all backends, and uses this data to route requests to the backend most likely to have relevant KV cache data.

kv_cache_index:
  enabled: true                      # Enable KV cache index (default: false)
  backend: memory                    # Index backend: "memory" (default) or "redis"
  max_entries: 100000                # Maximum prefix hash entries tracked
  entry_ttl_seconds: 600             # TTL for index entries in seconds
  scoring:                           # Scoring weights for backend selection
    overlap_weight: 0.6              # Weight for cache overlap signal (0.0-1.0)
    load_weight: 0.3                 # Weight for backend load signal (0.0-1.0)
    health_weight: 0.1               # Weight for backend health signal (0.0-1.0)
    min_overlap_threshold: 0.3       # Minimum overlap to activate KV-aware routing
    gpu_tier_weight: 1.0             # Tier multiplier for GPU-resident (hot) cached data
    storage_tier_weight: 0.6         # Tier multiplier for storage-offloaded (warm) cached data
  storage_offloading:                # Tiered storage awareness (GPU hot / storage warm)
    enabled: false                   # Enable storage tier tracking (default: false)
    treat_eviction_as_offload: true  # Treat eviction as offload to warm storage (default: true)
  event_sources:                     # continuum-kv-listener SSE endpoints
    - backend_name: "vllm-1"
      endpoint: "http://kv-listener.internal:7817/events/vllm-1"
      reconnect_interval_ms: 5000
    - backend_name: "vllm-2"
      endpoint: "http://kv-listener.internal:7817/events/vllm-2"
      reconnect_interval_ms: 5000
    # - backend_name: "vllm-3"         # Push mode: the listener dials the router's event bus
    #   transport: event_bus           # No endpoint; needs a role: kv_listener dynamo.event_bus.brokers[] entry

When using transport: event_bus or control-plane config sync (kv_routing), the following rules apply:

• Validation rules: With transport: event_bus, backend_name must be a valid event bus source id (1 to 64 bytes of [A-Za-z0-9_-]), must not be shared with any other source, and must omit endpoint. Violating any of these rules is a configuration load error.
• Runtime requirements and warnings: An event_bus source requires dynamo.event_bus to be enabled with a role: kv_listener entry in brokers[] that lists the backend under backends, and a binary compiled with the dynamo-event-bus feature (without the feature, the source is inert and logs a startup warning). Both config validate and startup logging emit non-fatal warnings when an event_bus source has no enabled dynamo.event_bus, when no kv_listener broker entry grants it, or when a kv_listener grant has no enabled event_bus source consuming it.
• Control-plane config sync (kv_routing): The Hub manages selection_strategy, prefix_routing.{enabled,max_prefix_length,load_factor_epsilon,virtual_nodes,anthropic_cache_control_injection}, and kv_cache_index.{enabled,max_entries,entry_ttl_seconds,scoring.*,storage_offloading.*,event_sources.<backend>.{endpoint,reconnect_interval_ms}} (for transport: sse sources). Meanwhile, prefix_routing.salt_echo, kv_cache_index.backend, and kv_cache_index.event_sources entries with transport: event_bus remain deployment-local and survive Hub applies, rebases, and restarts.

How It Works¶

1. Event consumption: Engines publish msgpack KV block events over ZMQ to continuum-kv-listener (both array-encoded batches from vLLM <= 0.28 and SGLang and map-encoded tagged-struct batches from vLLM 0.29+ are accepted); routers subscribe to the listener's SSE stream. Each SSE event reports a CacheCreated, CacheEvicted, CacheOffloaded, CacheReloaded, or CachePurged action with a prefix hash and token count.
2. Index update: Events update an in-memory (or Redis-backed) index mapping prefix hashes to backends with their cached token counts and storage tier (GpuHot or StorageWarm).
3. Scoring: On each request, the overlap scorer queries the index for the request's prefix key and computes a weighted score combining cache overlap (adjusted by tier multiplier), backend load, and backend health.
4. Routing decision: If the best score exceeds min_overlap_threshold, the scorer selects that backend. Otherwise, the configured selection strategy (e.g., PrefixAwareHash) is used as fallback.

Scoring Formula¶

final_score = overlap_weight * (raw_overlap * tier_multiplier)
            + load_weight * (1 - load_ratio)
            + health_weight * health_score

Where:

• raw_overlap = backend_token_count / max_token_count_across_backends (0.0 to 1.0)
• tier_multiplier = gpu_tier_weight for GPU-resident data, or storage_tier_weight for offloaded data
• load_ratio = in_flight_requests / max_in_flight (0.0 to 1.0)
• health_score = backend success_rate (0.0 to 1.0)

When storage_offloading.enabled is false, all data is treated as GpuHot and tier_multiplier always equals gpu_tier_weight (default: 1.0).

Configuration Options¶

Redis Backend¶

When backend: redis, the KV index uses the shared Redis connection from response_cache.redis. This enables multiple router instances to share the same index for consistent routing decisions across the fleet.

response_cache:
  redis:
    url: "redis://localhost:6379"
    pool_size: 8

kv_cache_index:
  enabled: true
  backend: redis    # Uses the Redis config from response_cache.redis

Admin Endpoints¶

• GET /admin/kv-index/stats: Index size, event rates, connection status, scoring distribution
• GET /admin/kv-index/backends: Per-backend cache state summary
• POST /admin/kv-index/clear: Clear the index (for debugging)

See Admin API for full response schemas.

Limitations¶

• Event sources only support vLLM-compatible SSE endpoints currently.
• Changing backend between memory and redis requires a restart.
• The KV index is an optional enhancement; its unavailability does not break routing.

Logging Section¶

Configures logging output:

logging:
  level: "info"                   # trace, debug, info, warn, error
  format: "json"                  # json, pretty, compact

level and format are read once, when the process starts. A hot reload that changes either reports logging under requires_restart; restart the router to apply it.

Log Levels:

• trace: Extremely verbose, includes all details
• debug: Detailed debugging information
• info: General operational information
• warn: Warning messages and potential issues
• error: Error conditions only

Log Formats:

• json: Structured JSON logging, one object per line with timestamp, level, target, and fields (recommended for production)
• pretty: Human-readable single-line text (good for development)
• compact: Abbreviated single-line text

Precedence:

• Level resolution: RUST_LOG overrides CONTINUUM_LOG_LEVEL, which overrides logging.level. When none of them is set, the default level is info.
• RUST_LOG filter directives: Accepts a full tracing filter directive such as continuum_router::proxy=debug,info. Unparseable directives are reported on stderr and ignored; empty values (such as RUST_LOG=${RUST_LOG:-} in Compose files) or strings with no directives are treated as unset.
• Preserving audit and dependency logs: Always include a bare default level in RUST_LOG (for example, continuum_router=debug,info). A filter that names only continuum_router targets (continuum_router=info) silences admin_audit, guardrail::audit, and dependency warnings because their targets live outside continuum_router. The router logs a startup warning if Admin audit logging is enabled but filtered out.
• Format fallback: logging.format has no environment variable override. If the configuration file is missing or fails to load, the process logs at info in json format, the same defaults a configuration without a logging section uses. When a file exists but fails to load (or the file named by --config is missing), the router logs a WARN line saying it fell back to these defaults, then reports the load error on stderr and exits. One-off subcommands (config, auth, metadata, mcp-serve, --migrate-config-file) do not read logging and always log to stderr at info in pretty format, for a person at a terminal.

Process-log queue¶

Log records are formatted on the thread that emits them and then handed to a bounded queue drained by one dedicated writer thread, so a slow or stalled log destination never blocks request processing. When the queue is full, new records are dropped and counted rather than waited on.

logging:
  level: "info"
  format: "json"
  queue_capacity: 8192            # records waiting for the writer (1 to 1,000,000)
  queue_capacity_bytes: 16777216  # bytes retained by queued records (65,536 to 1 GiB)

A record is accepted only when both limits allow it. A single formatted record is capped at 64 KiB; a longer one is cut on a character boundary and ends with [process log truncated]. Both fields require a restart, and an out-of-range value is rejected when the configuration loads.

Loss is visible in Prometheus rather than in the log itself:

A failed write to the destination stops process logging for the rest of the process lifetime; restart the Router to resume it. Queued records are flushed on graceful shutdown and on startup errors, within a one-second budget.

Request completion logging¶

Every request on the client API routes (the OpenAI-compatible and Anthropic endpoints such as /v1/chat/completions and /v1/models, not /health, /metrics, or /admin) emits one structured INFO log line (event=request_completed) when it finishes. The line carries request_id (matching the x-continuum-request-id response header), trace_id, method, route, status, duration_ms, model, backend (the backend that served the last recorded attempt, or the last one dialed when the request failed), key_id (the resolved API key id; never raw key material), ttft_ms, prompt_tokens, completion_tokens, total_tokens, and cached_tokens. A field the request never produced is left out: a request refused before dispatch has no backend, and a backend that reported no usage leaves the token fields out.

For a streaming request the line is written after the stream ends, so the token counts and ttft_ms (provider dispatch to the first streamed frame) are the final values. For a buffered reply ttft_ms is the time until the upstream response headers arrived. duration_ms runs until the client received the whole reply or disconnected. The request start line (Starting request with retry support) also carries the requested model.

Request logging middleware¶

A separate middleware writes a DEBUG line (Incoming request) for each request and, at TRACE, its headers. logging.access adjusts it:

logging:
  access:
    log_headers: true             # emit headers at TRACE level
    exclude_paths: [/metrics]     # replaces /health, /health/ready, /healthz
    completion_line: true         # one line per completed request
    success_level: info           # 2xx and 3xx
    client_error_level: warn      # 4xx
    server_error_level: error     # 5xx

The three level fields apply only while completion_line is on. The line is written for every non-excluded request, so a request on an API route produces both it and the event=request_completed line that the API path already writes. For a streamed reply the completion line is written when the response head is ready, so its duration stops at the headers and event=request_completed reports the stream end. A request refused before the handler (authentication, rate limiting, validation) is reported by event=request_rejected instead, and an excluded path writes no completion line.

Changes require a restart. Request and response bodies are never written by this middleware; body capture is tracked under the opt-in prompt-logging epic (#2294).

Pre-backend rejection logging¶

Requests rejected before a backend is selected (auth, rate_limit, policy, admission, guardrail_input, or routing) emit one structured WARN log line (event=request_rejected) carrying trace_id, x_request_id, client_ref, request_id (the same value as the x-continuum-request-id response header), method, route, status, error.type, error.code, limit_type, stage, key_id (when resolved; never raw key material), and latency.

Log cap and summary lines¶

To bound log volume during an unauthenticated or rate-limited flood, logging.rejections.max_per_minute (default 600) caps how many event=request_rejected lines are emitted in each 60-second window. Each suppressed rejection increments request_rejected_log_suppressed_total.

logging:
  rejections:
    max_per_minute: 600

A window that reaches the cap produces two WARN lines with event=request_rejected_summary, told apart by phase:

• phase="cap_reached": written once, at the first suppressed rejection of the window. It carries max_per_minute and window_secs and marks the start of suppression.
• phase="window_closed": written once after the window ends. It reports how many lines the window suppressed and for which keys, using the fields below.

The key labels in suppressed_by_key follow the same rules as the request_rejected_total labels described below, including the other, unattributed, and invalid buckets, and each window tracks at most 256 key ids. Unlike the counter, a window admits key ids first come, first served: key ids beyond the first 256 in a window go to other, and nothing is evicted. The characters , : = " \ and % in a key id are percent-encoded (, becomes %2C), so the field always splits cleanly on , and the last :.

The router has no background timer for this line. It writes the window_closed summary when the next request of any kind arrives after the window ends, health checks included, so on an idle router the line appears later than the end of the window.

Per-key rejection counter¶

Every pre-backend rejection is also counted per API key, before the log cap applies, so the counts stay exact while log lines are suppressed. With metrics enabled, the counter is exported as request_rejected_total{stage, key_id}. To keep the label set bounded, the router tracks the 256 most recently rejected key ids. When a rejection arrives for a new key id while 256 are tracked, the least recently rejected key is evicted: its counts move into key_id="other" and its own series are removed, so totals summed over key_id stay exact. Rejections with no resolved key (for example an unknown key, or a rate-limit 429 that fires before authentication) are counted under key_id="unattributed", and key ids longer than 128 bytes, containing characters outside printable ASCII, or equal to a bucket name under key_id="invalid". The labels carry key ids only, never key material.

The Continuum Hub usage ledger still receives only attempts that reached a backend. When the router is enrolled with Continuum Hub, the heartbeat also reports rejections of Hub-issued keys per key and stage in rejections_by_key. That list holds at most 32 keys: the first 32 Hub keys rejected after process start keep their rows until the router restarts, so each row only grows, and rejections of any other Hub key are summed in rejections_by_key_other. Rejections of local keys, AppProxy keys, and requests with no resolved key reach the Hub only through the aggregate rejected_requests counter, because the Hub cannot interpret those key ids.

Alerting on suppression¶

The summary lines are for reading after the fact; alert on the counters instead. The first expression below is true while the router has suppressed any rejection log line in the last five minutes. The second returns exact per-key counts for the same period, including the rejections whose log lines were suppressed. The exception is key_id="other": an eviction moves the evicted key's lifetime count into it at once, so its increase() and rate() jump on an eviction and are not per-period counts. Alert on key_id!="other" series instead.

# Alert: rejection logs are being suppressed
rate(request_rejected_log_suppressed_total[5m]) > 0

# Rejections per key and stage over the last 5 minutes
sum by (key_id, stage) (increase(request_rejected_total[5m]))

File output¶

Logs go to stdout (stderr in stdio mode) by default, which is the right choice when Docker, Kubernetes, or journald owns persistence and rotation. The optional file block writes to rotating files instead:

logging:
  level: "info"
  format: "json"
  file:
    directory: /var/log/continuum-router
    filename: continuum-router-{date}-{time}-{pid}-{sequence}.log
    rotation:
      max_bytes: 104857600        # 100 MiB (1 MiB to 1 TiB)
      max_files: 10               # 1 to 10,000
      max_age_days: 0             # 0 to 36,500; 0 disables the age cap

Files are created without overwriting an existing name and, on Unix, with mode 0600. They contain plain text with no terminal colour codes. The writer flushes at least once per second and on rotation, but does not fsync, so a host crash can lose the most recent lines. If a new file cannot be created during rotation (for example on a full disk) or old files cannot be pruned, the Router reports it on stderr and keeps writing to the current file.

Do not combine this rotation with an external rename-based logrotate rule for the same files: the Router keeps its current file open. One-off commands such as config validate and mcp-serve always log to stderr and never touch the file sink.

The design, failure policy, and benchmark are described in Log System Architecture.

Prompt Logging Section¶

Configures opt-in capture of prompt and completion bodies. The section is off by default. While enabled is false, no request path reads, stores, or exports a body, and metadata-only logging, usage, and tracing are unchanged.

To enable and operate the feature, see the Prompt Logging guide.

While enabled is true, each POST /v1/chat/completions, POST /v1/responses, or POST /anthropic/v1/messages request that passes the consent gate appends one JSONL record to a file under directory. The record carries the prompt the client sent, the completion and tool calls the backend produced, the usage block, the consent basis, and one of these outcomes:

• ok: the reply was served.
• refused: a guardrail blocked the request or the reply; the record carries the output the backend generated, when the route kept it (a buffered chat reply and any stream; a buffered /v1/responses, /anthropic/v1/messages, or disaggregated refusal has no output).
• backend_error: no backend answered.
• client_cancelled: the client left a stream early; the record carries the partial reply.
• stream_errored: an upstream error or timeout after the stream began.

A request that fails over to another model writes one record, for the hop that answered. The A2A bridge onto chat completions is captured the same way. /v1/completions, embeddings, and ACP are not captured yet. A response-cache hit writes no record, nor does a request refused before dispatch by validation or request policy, nor a buffered request the client cancelled while the router read an internal stream.

Disaggregated serving is captured the same way, buffered and streamed, and a fallback to unified serving writes the record of the unified reply. A request the orchestrator refuses, including a Dynamo shim's direct-mode refusal, is backend_error with the HTTP status as its error.

prompt_logging:
  enabled: false
  directory: ./prompt-logs        # created 0700; files 0600
  default_consent: allowed        # allowed, denied
  capture:
    prompts: true
    completions: true
    tool_calls: true
  redaction:
    patterns: []                  # regular expressions
    masked_replacement: "[redacted]"
  retention:
    max_bytes: 104857600          # 100 MiB per file
    max_files: 10
    max_age_days: 7
  export:
    hub_push: false
    sinks: []

Each export.sinks entry selects its destination with type:

sinks:
  - type: webhook
    endpoint: "https://logs.example.com/ingest"
    headers_env:
      Authorization: COLLECTOR_AUTH
  - type: langfuse
    endpoint: "https://langfuse.example.com"
    public_key_env: LANGFUSE_PUBLIC_KEY
    secret_key_env: LANGFUSE_SECRET_KEY
  - type: s3
    bucket: prompt-logs
    prefix: router-1/

Validation¶

The router refuses to start, and config validate reports an error, in these cases:

• An empty directory.
• enabled: true with all three capture scopes set to false.
• A redaction.patterns entry that is not a valid regular expression or that matches the empty string. This check also runs while enabled is false.
• A retention bound of 0. Retention is always finite, so 0 does not mean unlimited; set a large value for long retention.
• A sink endpoint that is not an http or https URL with a host, or that embeds credentials or carries a query string or fragment.
• An s3 sink whose bucket is not a bucket name (a key:secret@bucket shape included), or whose prefix or region is malformed.
• A sink *_env field or headers_env value that is not an environment variable name, or a sink bound outside its range.
• A redaction.masked_replacement containing $. A reference such as $0 would write the matched text back into the record, and the $$ escape does not lift this rule.

A redaction.patterns error names the entry by index and gives the regular expression error kind, never the pattern text.

A sink endpoint may hold a ${VAR} reference. With the variable unset, config validate reports a warning and skips the URL checks, and the router refuses to start. With it set, the checks run on the resolved value.

export.hub_push: true loads normally and logs a warning in a binary built without the control-plane feature, or when control_plane is not enabled, because nothing then sends to a Hub. So does an s3 sink in a binary built without the prompt-log-s3 feature.

While enabled is true, a sink whose endpoint uses http:// on a host that does not look private also draws a warning from config validate and at startup, because prompt content and credentials would travel in clear text. The sink still runs. Loopback, private, and link-local addresses, localhost, single-label names, and .svc names draw no warning.

Changing any field requires a restart: a hot reload reports prompt_logging under requires_restart. The writer, the redactor, and the open file are built at startup, and the consent gate reads the startup section.

Record Files¶

Records are written by one dedicated thread, never by the request. The request path redacts the record once and hands it to a bounded queue, so unredacted text never reaches the file, a log line, or an error. Redaction applies the built-in credential patterns, then redaction.patterns, and replaces each match literally with redaction.masked_replacement.

The writer is fail-open. prompt_log_records_total{result} counts every record the capture path produced:

• written: the writer appended the record to the file.
• queue_full: the queue was full and the record was dropped.
• sink_error: the write failed (a full disk, a removed directory) and the record was dropped; one ERROR line without content reports the failure.

A dropped record never changes the request's outcome. On shutdown the queue is drained and the file flushed after the request drain.

Truncated Fields¶

A record's truncated list names each field that does not hold everything the request or reply carried:

• Byte cap: model, prompt, completion, and the tool calls together are each cut at 256 KiB.
• Early stream end: a stream that ended as client_cancelled, stream_errored, or refused stopped at an arbitrary point, so its completion and each tool call's arguments are cut there.
• Masking bound: masking keeps at most 1 MiB for one field, and for the tool calls together. A pattern that matches its own replacement can reach this bound, and the rest of the text is not written.

A cut trims the text back to its last whitespace, at most 4 KiB, so a credential crossing the cut leaves no prefix that its pattern would miss. Text with no whitespace in that tail loses all of it, so a short reply can be empty with its field still listed.

Age Retention¶

While the router runs, no record stays on disk past retention.max_age_days. A file can only be removed whole, so the bound follows from these rules:

• Timer: the writer thread applies retention hourly with or without traffic, and also at startup. A stopped router removes nothing until it starts again.
• Roll: the active file rolls once its first record is one eighth of max_age_days old. An empty file does not roll.
• Removal: a file is removed when its last write is older than max_age_days, less one eighth, less two hours.

A record is therefore removed between about seven eighths of the bound and the bound: about 6 to 7 days under the default. Traffic creates one file per eighth of the bound, so retention.max_files needs to be at least 9 for age to be the bound that applies.

A streamed reply is recorded once, when the stream ends, from the output the router relayed. The output is read ahead of the guardrail output gate, so a refused stream record carries the text the gate withheld from the client. A per-request fallback list, a configured chain, and a mid-stream hop all yield one record, for the hop that ended the request, and the record's completion is that hop's output.

Export Sinks¶

Each sink tails the record files independently and delivers batches off the request path. A sink runs only while enabled is true; with the section off, no export code runs and no cursor is written. The Hub push is a further task of the same loop, with its own cursor and a stricter consent rule; see Hub push.

A credential is always an environment variable name, read once at startup. A variable that is unset or empty stops the router from starting, and config validate warns about it first. A binary without prompt-log-s3 reads no variable for an s3 sink, which is inert there. The message names the variable, never a value.

Delivery¶

Delivery is at least once. Each sink keeps a cursor (file sequence and byte offset) in .export-cursor-<id>.json under directory, created 0600, and moves it after a batch is delivered. The <id> is a hash of the sink's type and destination, so reordering sinks keeps each cursor, while changing a destination (including a token in a webhook URL) makes it a new sink. With cursor_id set, the <id> is <type>-<cursor_id> and a changed destination keeps the cursor; adding it to an existing sink starts a new cursor once, following backfill.

Where a sink starts depends on its cursor file:

• No cursor file, backfill: false (the default): the sink starts at the end of the retained records and sends only what is written afterwards. The start position is stored at once, so a restart still sends records written while the router was stopped. Deleting a cursor file makes the sink new again.
• No cursor file, backfill: true: the sink starts at the oldest file retention still holds and sends every retained record, at once and in batches.
• A cursor file that cannot be read (for example EMFILE, EIO, or EACCES): the sink logs one warning and retries every interval_seconds until the read succeeds or the router shuts down. It sends nothing meanwhile, and it never moves to the oldest file or the tail, because the stored position is likely intact.
• A cursor file that holds no valid cursor, backfill: false: the sink logs a warning and starts at the end of the retained records, as it does after a deleted cursor. Records written after the lost position and before the restart are not sent.
• A cursor file that holds no valid cursor, backfill: true: the sink logs a warning and restarts from the oldest file, so records it already sent are delivered again.

If a cursor file stays unreadable when the router opens the record files, the router writes no records until the next start: the next file number must be above every cursor, and that cursor's number is unknown. Requests are still served. A file that holds no valid cursor does not affect startup.

backfill has no effect on a sink that already has a cursor. Export applies the current redaction.patterns and re-checks each record's key consent, as described in Consent withdrawal. It does not re-evaluate capture scope, which has two consequences:

• Backfilled records include those captured under an earlier capture scope.

• A sink with an existing cursor still delivers its backlog after a capture scope narrows. To stop that delivery, remove the sink from export.sinks.

• Failure: the batch stays on disk and is retried with exponential backoff, from interval_seconds up to max_backoff_seconds. A request is never affected, and a sink holds one batch in memory however long the outage lasts. The first failure logs a warning and each attempt counts as failed. With the default on_permanent_failure: block, a sink whose destination refuses a record permanently, such as with a 4xx response, delivers nothing further until the operator fixes it or retention prunes the file.

• Skipping a refusal: with on_permanent_failure: skip, an eligible status drops the rest of the batch and moves the cursor. See the prompt logging guide.
• Unreadable cursor at startup: after five retries, the router serves requests with no capture and no export until the next start, and sets prompt_log_capture_disabled{reason="cursor_unreadable"} to 1. A retention-only timer continues applying max_age_days and max_files without creating a record file. See the prompt logging guide.
• Retention: retention does not wait for a sink. If it removes a file a sink has not read, the cursor jumps to the next file, and pruned counts the jump. Size retention for the longest outage you want to survive.
• Masking: every record passes redaction.patterns again on read, so a pattern added after capture also covers older files.
• Malformed lines: a line that is not a record is skipped and counted. A partial line at the end of the file being written is left until it completes.

Destinations¶

• webhook: a POST of a JSON array of records.
• langfuse: a POST to {endpoint}/api/public/otel/v1/traces as OTLP JSON, with x-langfuse-ingestion-version: 4. Each record becomes one complete span of observation type generation, with input, output, model, and token usage. traceId and spanId are derived from request_id. Spans Langfuse reports in partialSuccess.rejectedSpans count as rejected and are not retried. The router does not use /api/public/ingestion, which Langfuse deprecated and sunsets on Langfuse Cloud on 2026-11-16.
• s3: one JSONL object per batch, named <prefix>/prompt-log-<sequence>-<offset>.jsonl, so a redelivered batch overwrites its earlier upload. Needs the prompt-log-s3 build feature, which official release binaries carry; a binary without it warns that the sink is inert.

Note: Langfuse v4 does not reliably deduplicate a span it receives twice, so a redelivered record can appear twice there even with the same ids.

Metrics¶

prompt_log_export_backlog_bytes{sink} reports the record bytes on disk a sink has not delivered. prompt_log_export_total{sink,result} counts by result. sink is the sink's type, with -1, -2, and so on for a further sink of the same type.

Log lines name the sink's label, a count, or a failure kind, never a record, an endpoint, or a credential.

Masked Output¶

A webhook URL can carry a token in its path, and a redaction pattern can quote a secret, so validation cannot detect either one. Sink endpoint values and redaction.patterns are masked in these outputs:

• Admin config responses, exports, and history. An unchanged placeholder imports back as the running value.
• config diff and the MCP diff_configs and resolve_config tools.
• config show --resolved. Its output does not load until the real values are restored. Plain config show prints the file as written.