Configuration Guide¶

This guide documents how to configure Continuum Router. The router supports multiple configuration methods with a clear priority system for different deployment scenarios.

Configuration sections:

• Server & Backends: Server settings, backend providers, and connection options
• Health & Caching: Health checks, request settings, retry, caching, and logging
• Security & Admin: API keys, authentication, WebUI, admin endpoints, and ACP
• Budgets: Data-plane long-window token and cost quotas
• Advanced: Global prompts, model metadata, hot reload, tracing, load balancing, rate limiting
• Examples: Configuration examples and Rust Builder API

Configuration Methods¶

Continuum Router supports four configuration methods:

1. Configuration File (YAML) - Recommended for production
2. Environment Variables - Ideal for containerized deployments
3. Command Line Arguments - Useful for testing and overrides
4. Rust Builder API - Type-safe programmatic configuration for library usage

Configuration Discovery¶

The router automatically searches for configuration files in these locations (in order). At each location, the .yaml extension is tried first, then .toml:

1. Path specified by --config flag
2. ./config.yaml or ./config.toml (current directory)
3. ~/.config/continuum-router/config.yaml or config.toml (user config directory)
4. /etc/continuum-router/config.yaml or config.toml (system config directory)

Configuration Priority¶

Configuration is applied in the following priority order (highest to lowest):

1. Command-line arguments (highest priority)
2. Environment variables
3. Configuration file
4. Default values (lowest priority)

This allows you to: - Set base configuration in a file - Override specific settings via environment variables in containers - Make temporary adjustments using command-line arguments

Configuration File Format¶

Complete Configuration Example¶

# Continuum Router Configuration
# Generate the full annotated sample with: continuum-router --generate-config

# Server configuration
server:
  # bind_address accepts a single string or an array of addresses
  # TCP format: "host:port", Unix socket format: "unix:/path/to/socket"
  bind_address: "0.0.0.0:8080"          # Single address
  # bind_address:                        # Or multiple addresses:
  #   - "0.0.0.0:8080"                   #   TCP on all interfaces
  #   - "unix:/var/run/router.sock"      #   Unix socket (Linux/macOS, Windows 10 1809+)
  # socket_mode: 0o660                   # Optional: Unix socket file permissions
  workers: 4                             # Number of worker threads (0 = auto-detect)
  connection_pool_size: 100              # Max idle connections per backend
  # max_concurrent_requests: 64          # In-flight request ceiling (restart required)
  # max_stream_buffer_bytes: 4294967296  # Process-wide SSE/relay buffer budget, default 4 GiB (restart required)
  # shutdown_drain_timeout: "20s"        # Planned-shutdown drain deadline (restart required)
  # explain_response_header: false       # Emit x-continuum-route on API responses (restart required)
  # Shared outbound HTTP client tuning. Optional, and every field defaults to
  # the value shown.
  # The client is built once at startup, so each field requires a restart.
  # http_client:
  #   pool_idle_timeout: "300s"          # How long an idle pooled connection is kept
  #   pool_max_idle_per_host: 64         # Idle connections retained per origin
  #   tcp_keepalive: "30s"               # TCP keep-alive probe interval
  #   http2_keep_alive_interval: "15s"   # HTTP/2 PING interval
  #   http2_keep_alive_timeout: "5s"     # How long a PING may go unanswered
  #   http2_keep_alive_while_idle: true  # Keep pinging idle connections

# Model metadata configuration (optional)
model_metadata_file: "model-metadata.yaml"  # Base (layer zero) metadata file
model_metadata_dirs:                        # Extra model-metadata.d drop-in directories
  - "/etc/router/metadata"                  # Applied after the conventional ones, in order

# Load balancing strategy: RoundRobin (default), WeightedRoundRobin,
# LeastLatency, Random, ConsistentHash, PrefixAwareHash, EngineLoad, LeastBusy,
# LeastCost
selection_strategy: RoundRobin

# Backend configuration
backends:
  # Native OpenAI API with built-in configuration
  - name: "openai"
    type: openai                         # Use native OpenAI backend
    api_key: "${CONTINUUM_OPENAI_API_KEY}"  # Loaded from environment
    org_id: "${CONTINUUM_OPENAI_ORG_ID}"    # Optional organization ID
    weight: 3
    models:                              # Specify which models to use
      - gpt-5.6-sol
      - gpt-5.6-terra
      - o3-mini
      - text-embedding-3-large
    retry_override:                      # Backend-specific retry settings (optional)
      max_attempts: 5
      initial_delay: "200ms"
      max_delay: "30s"
      backoff_multiplier: 2.0
      jitter: true
      retryable_status_codes: [429, 502, 503, 504]
      retryable_errors: [ConnectionError, TimeoutError]
      timeout: "60s"

  # Generic OpenAI-compatible backend with custom metadata
  - name: "openai-compatible"
    url: "https://custom-llm.example.com"
    weight: 1
    models:
      - "gpt-5.6-sol"
      - "gpt-5.6-luna"
    model_configs:                       # Enhanced model configuration with metadata
      - id: "gpt-5.6-sol"
        aliases:                         # Alternative IDs that share this metadata (optional)
          - "gpt-5.6"
        metadata:
          display_name: "GPT-5.6 Sol"
          summary: "Frontier GPT-5.6 model for complex professional work"
          capabilities: ["chat", "vision", "code", "reasoning", "tool"]
          knowledge_cutoff: "2026-02"
          pricing:                       # USD per 1M tokens
            input_tokens: 5.0
            output_tokens: 30.0
          limits:
            context_window: 1050000
            max_output: 128000

  # Ollama local server with automatic URL detection
  - name: "local-ollama"
    type: ollama                         # Defaults to http://localhost:11434
    weight: 2
    models:
      - "llama2"
      - "mistral"
      - "codellama"

  # vLLM server
  - name: "vllm-server"
    type: vllm
    url: "http://localhost:8000"
    weight: 1
    # Models will be discovered automatically if not specified
    # Models with namespace prefixes (e.g., "custom/gpt-5.6-sol") will automatically
    # match metadata for base names (e.g., "gpt-5.6-sol")
    model_configs:
      - id: "qwen3.8-27b"
        metadata:
          artifact:                      # Weights this backend serves (optional)
            digest: "sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"
            revision: "4c2e9a1"
            source: "hf://Qwen/Qwen3.8-27B-FP8"
          draft_model:                   # Speculative-decoding draft (optional)
            name: "qwen3.8-0.6b"

  # Google Gemini API (native backend)
  - name: "gemini"
    type: gemini                           # Use native Gemini backend
    api_key: "${CONTINUUM_GEMINI_API_KEY}" # Loaded from environment
    weight: 2
    models:
      - gemini-3.1-pro-preview
      - gemini-3-flash-preview
      - gemini-2.5-pro
      - gemini-2.5-flash

# Health monitoring configuration
health_checks:
  interval: "30s"                        # How often to check backend health
  timeout: "5s"                          # Timeout for health check requests
  unhealthy_threshold: 3                 # Failures before marking unhealthy
  healthy_threshold: 2                   # Successes before marking healthy
  endpoint: "/health"                    # Endpoint used for health checks
  warmup_check_interval: "1s"            # Accelerated interval while a backend warms up (HTTP 503)
  max_warmup_duration: "300s"            # Max time in accelerated warmup mode
  block_startup: true                    # Gate listener binding on pre-warm and the initial health check
  prewarm_timeout: "10s"                 # Per-request bound on the startup pre-warm probes (restart to change)

# Request handling and timeout configuration
timeouts:
  connection: "10s"                      # TCP connection establishment timeout
  request:
    standard:                            # Non-streaming requests
      first_byte: "30s"                  # Deprecated and inert; `total` is the real budget
      total: "180s"                      # Total request timeout (3 minutes)
    streaming:                           # Streaming (SSE) requests
      first_byte: "120s"                 # Enforced deadline for the first SSE chunk
      chunk_interval: "30s"              # Max time between later chunks
      total: "600s"                      # Total streaming timeout (10 minutes)
    image_generation:                    # Image generation requests (DALL-E, etc.)
      first_byte: "60s"                  # Deprecated and inert, like standard.first_byte
      total: "180s"                      # Total timeout (3 minutes default, cap 600s)
    model_overrides:                     # Model-specific timeout overrides
      gpt-5-latest:
        streaming:
          total: "1200s"                 # 20 minutes for GPT-5
      gpt-5.6-sol:
        streaming:
          total: "900s"                  # 15 minutes for GPT-4o
  # Validation ceilings for this whole section. Pinned at startup: hot reload,
  # the Admin API, and config sync reject a candidate that changes them.
  # limits:
  #   max_standard_timeout: "900s"       # Default 540s, absolute ceiling 1080s
  #   max_retry_timeout: "90s"           # Default 30s, absolute ceiling 120s

# Global retry and resilience configuration
retry:
  max_attempts: 3                        # Maximum retry attempts
  initial_delay: "100ms"                 # Initial delay between retries
  max_delay: "10s"                       # Maximum delay between retries
  backoff_multiplier: 2.0                # Exponential backoff multiplier
  jitter: true                           # Add random jitter to delays
  retryable_status_codes: [429, 502, 503, 504]
  retryable_errors: [ConnectionError, TimeoutError]
  timeout: "30s"                        # Total retry budget across all attempts
  prefer_untried_backends: false        # Retry onto a backend this request has not dialed yet

# Logging configuration
logging:
  level: "info"                         # Log level: trace, debug, info, warn, error
  format: "json"                        # Log format: json, pretty

# Files API configuration
files:
  enabled: true                         # Enable/disable Files API endpoints
  max_file_size: 536870912              # Maximum file size in bytes (default: 512MB)
  storage_path: "./data/files"          # Storage path for uploaded files (supports ~)
  retention_days: 0                     # Startup retention sweep in days (0 = keep forever)
  metadata_storage: persistent          # Metadata backend: "memory" or "persistent" (default)
  cleanup_orphans_on_startup: false     # Auto-cleanup orphaned files on startup

  # Authentication and authorization
  auth:
    method: api_key                     # "none" or "api_key" (default)
    required_scope: files               # API key scope required for access
    enforce_ownership: true             # Users can only access their own files
    admin_can_access_all: true          # Admin scope grants access to all files

  # OpenAI backend file handling strategy
  openai:
    strategy: base64                    # "base64" (default) or "upload"
    fallback_to_base64: true            # Fall back to base64 if upload fails
    cache_capacity: 1000                # Max cached file ID mappings
    cache_ttl_secs: 86400               # Cache TTL in seconds (default: 24h)

  # Per-endpoint rate limiting for file operations
  rate_limiting:
    enabled: false                      # Enable Files API rate limiting
    uploads_per_minute: 10              # Max file uploads per minute
    downloads_per_minute: 100           # Max file downloads per minute
    operations_per_minute: 1000         # Max list/get/delete operations per minute

# Distributed tracing configuration
tracing:
  enabled: true                         # Enable/disable distributed tracing
  w3c_trace_context: true               # Support W3C Trace Context (traceparent header)
  headers:
    trace_id: "X-Trace-ID"              # Header name for trace ID
    request_id: "X-Request-ID"          # Header name for request ID
    correlation_id: "X-Correlation-ID"  # Header name for correlation ID

# Circuit breaker configuration
circuit_breaker:
  enabled: true                         # Protect ordinary proxy dispatch and expose Admin controls
  failure_threshold: 5                  # Consecutive failures to open the circuit
  failure_rate_threshold: 0.5           # Failure rate (0.0-1.0) to open the circuit
  minimum_requests: 10                  # Minimum requests before rate evaluation
  timeout: "60s"                        # Time before attempting recovery (half-open)
  half_open_max_requests: 3             # Trial requests allowed in half-open state
  half_open_success_threshold: 2        # Successes required to close the circuit

# Ordinary proxy traffic records outcomes and routes around open circuits.
# Health checks, retry, and fallback remain complementary layers.

# Rate limiting configuration
rate_limiting:
  enabled: true                         # Enable rate limiting
  storage: memory                       # "memory" or "redis"
  limits:
    per_client:
      requests_per_second: 10
      burst_capacity: 20
    global:
      requests_per_second: 1000
      burst_capacity: 2000

# Admin API configuration
admin:
  max_backend_name_length: 64             # Maximum backend name length (1-256, default: 64)
  auth:
    method: bearer                         # Auth method: none, bearer, basic, ip_whitelist, api_key
    bearer_token: "${ADMIN_TOKEN}"         # Admin authentication token
  stats:
    enabled: true                          # Enable/disable stats collection
    retention_window: 24h                  # Ring-buffer retention for windowed queries
    token_tracking: true                   # Parse response bodies for token usage
    persistence:
      enabled: true                        # Enable stats persistence across restarts
      path: ./data/stats.json              # File path for the snapshot
      snapshot_interval: 5m                # How often to write periodic snapshots
      max_age: 7d                          # Discard snapshots older than this on startup
    # Per-record usage export for GET /admin/usage/export (off by default).
    # With persistence on, each record adds about 300 bytes to the snapshot
    # (about 300 MB at 1,000,000), so size max_records to the audit window.
    usage_export:
      enabled: false                       # Needs stats.enabled too; disabling drains the records
      retention: 7d                        # Unparseable or zero is a load error
      max_records: 1000000                 # 1,000 to 10,000,000; oldest evicted first

# Metrics and monitoring configuration
metrics:
  enabled: true                         # Enable Prometheus metrics collection
  path: "/metrics"                      # Metrics endpoint path
  port: 9090                            # Dedicated metrics-only listener (0 = off); restart to change
  # bind_address: "0.0.0.0"             # Its host; default: first TCP server.bind_address host
  auth:
    method: bearer
    bearer_token: "${METRICS_TOKEN}"    # Omitting auth makes the endpoint return 401

# Watch-only NVIDIA Dynamo worker discovery (build feature `dynamo`,
# compiled into official release binaries). The router watches etcd, the
# DynamoWorkerMetadata resources and EndpointSlices of one namespace, or its
# operator-labelled Pods, for the workers Dynamo registers and never writes,
# deletes, or leases anything. Nothing here needs a restart:
# `load.*`, `shim.*`, and `kv.*` reload immediately, and the rest
# (`source`, `endpoints`, `tls`, `auth`, and every `kubernetes.*` field)
# replaces the watcher, which redials and resyncs in full (`kubernetes.model`
# and `kubernetes.models` are ignored by `source: kubernetes` and editing them
# changes nothing there). `event_bus` runs the in-cluster event broker
# listener under its own supervisor (see Dynamo event broker listener below). See
# architecture/dynamo-integration.md.
dynamo:
  enabled: false                        # Master switch; inert without the `dynamo` build feature
  discovery:
    source: etcd                        # `etcd` (default), `kubernetes` (DynamoWorkerMetadata + EndpointSlices; load and capacity like etcd), or `kubernetes_pods` (Pods alone; no load or capacity: round-robin dispatch)
    etcd:
      endpoints: ["http://etcd:2379"]   # http(s) URLs or bare host:port; reloads live
      namespace: "dynamo"               # Must match the workers' DYN_NAMESPACE
      # namespaces: ["prod", "canary"]  # Or several graphs (at most 16), each bound to its own `shim.pools` entry; adding or removing one keeps the others' workers
      watch_reconnect_backoff:
        initial: "500ms"                # First reconnect delay after a dropped watch
        max: "30s"                      # Ceiling the delay doubles up to
    # kubernetes:                       # Required with `source: kubernetes` or `kubernetes_pods`; validated whenever present
    #   namespace: "dynamo"             # Kubernetes namespace of the fleet (get/list/watch on dynamoworkermetadatas + endpointslices, or on pods)
    #   dynamo_namespace: "dynamo"      # The workers' DYN_NAMESPACE (their nvidia.com/dynamo-namespace label and record namespace)
    #   dynamo_namespaces: ["prod", "canary"]  # Or several Dynamo namespaces in this Kubernetes namespace; exclusive with `dynamo_namespace`; `models` entries then name their `dynamo_namespace`
    #   model: "Qwen/Qwen3-32B"         # Served model name; `kubernetes_pods` needs it or `models` (Pods carry no card), `kubernetes` ignores both. Or, exclusive with it:
    #   models:                         # One entry per nvidia.com/dynamo-component; unlisted components are skipped
    #     - {component: "VllmWorker-decodeheavy", model: "solar-pro4"}
    #     - {component: "VllmWorker-prefillheavy", model: "solar-pro4-longcontext"}
    #   watch_reconnect_backoff: {initial: "500ms", max: "30s"}
  load:
    max_staleness: "3s"                 # Freshness window for a worker's last kv_metrics sample (100ms-5m)
    transport: direct                   # direct | broker: how kv_metrics and kv-events subscriptions reach publishers
  shim:
    backends: []                        # `dynamo.frontend --router-mode direct` replicas, by backends[].name; needs disaggregated_serving.enabled
    # pools:                            # With several namespaces: one pool per namespace instead of `backends`; a pin only ever reaches its own namespace's frontend
    #   - {namespace: "prod", backends: [dynamo-frontend-prod]}
    #   - {namespace: "canary", backends: [dynamo-frontend-canary]}
    model_map: {}                       # Client model id -> Dynamo display_name for candidate matching only; the body is forwarded unchanged
  kv:                                   # KV-overlap-aware prefill selection (build feature `dynamo-kv`); all fields reload immediately
    enabled: false                      # Subscribe to kv-events, index blocks, tokenize with the fleet's artifacts, credit prefix overlap
    # artifact_cache_dir: /var/cache/continuum-router/dynamo-kv-artifacts  # Default: <user cache dir>/continuum-router/dynamo-kv-artifacts; blake3-verified
    allow_http_artifacts: true          # Fetch a card's http(s):// artifact (a worker's own metadata endpoint)
    allow_hf_artifacts: false           # Fetch a card's hf:// artifact from the Hub (public egress; HF_ENDPOINT / HF_TOKEN honored)
    pinned_artifacts: []                # blake3 checksums of the files the router may load; nothing is loaded, compiled, or rendered until pinned (validate warns when empty)
    # pinned_artifacts: ["blake3:<64 hex>", ...]  # from `b3sum` over the reviewed tokenizer.json / tokenizer_config.json / chat_template.jinja
    overlap_score_credit: 1.0           # Blocks of credit per block of prefix overlap (finite, >= 0)
    prefill_load_scale: 1.0             # Multiplier on the adjusted prefill term (finite, >= 0)
    router_temperature: 0.0             # 0 = lowest cost wins; > 0 softmax-samples over costs (<= 100)
    send_token_data: false              # Send the rendered token ids as nvext.token_data (couples the tokenizer version to the fleet's)
    conditional_disaggregation:         # Skip remote prefill for a short effective ISL; refused by a Dynamo 1.5.x direct-mode frontend
      enabled: false
      policy: isl_bounding              # isl_bounding | prefill_load | isl_or_load
      eff_isl_threshold: 2048           # Effective ISL must be below this many tokens
      eff_isl_ratio_threshold: 0.7      # Effective/raw ISL must be below this (0-1)
      # prefill_busy_threshold: 0.8     # prefill_load / isl_or_load: busy above this fraction of max_num_batched_tokens
      # decode_busy_threshold: 0.9      # Veto a bypass onto a busy decode worker, or one with no signal
  event_bus:                            # In-cluster event broker listener (build feature `dynamo-event-bus`); independent of `enabled`
    enabled: false                      # Starts the TLS listener continuum-event-broker dials out to; off drains every session
    listen: "0.0.0.0:7443"              # Required when enabled; its own port, never a server.bind_address port
    tls:                                # Required when enabled; TLS 1.3, ALPN h2, plaintext refused
      cert: "/etc/continuum-router/event-bus/tls.crt"
      key: "/etc/continuum-router/event-bus/tls.key"
      # client_ca: "/etc/continuum-router/event-bus/client-ca.crt"  # mTLS: a certificate whose DNS SAN is a broker id authenticates as it
    brokers:
      - id: "broker-a.dynamo"           # Lowercase [a-z0-9.-], <= 63 bytes; must equal the broker's hello.client_id
        token_hash: "sha256:0000000000000000000000000000000000000000000000000000000000000001"  # Placeholder: printf %s "$TOKEN" | sha256sum; never the token
        namespaces: ["dynamo"]          # Dynamo namespaces this broker may declare; empty accepts any (validate warns)
      # - id: "kv-listener-a"           # continuum-kv-listener in push mode
      #   role: kv_listener             # broker (default) | kv_listener
      #   token_hash: "sha256:0000000000000000000000000000000000000000000000000000000000000003"  # Placeholder
      #   backends: ["vllm-1"]          # kv_cache_index event_sources backend_name values this listener may announce
    max_sessions: 64                    # Router-wide welcomed sessions (1-1024)
    max_streams_per_session: 1024       # Subscriptions per session (1-1024)
    max_session_buffer_bytes: 67108864  # Per-session byte budget; 16 KiB per stream per direction with these defaults
    heartbeat_interval: "10s"           # 1s-30s
    idle_timeout: "30s"                 # >= 2 x heartbeat_interval, <= 300s

# Continuum Hub control-plane agent (requires the `control-plane` build feature)
control_plane:
  enabled: false                        # Master switch
  hub_url: "https://hub.example.com"    # Required when enabled
  enrollment_token: "${CONTINUUM_HUB_ENROLLMENT_TOKEN}"
  state_file: "./data/control-plane/state.json"
  heartbeat_interval_secs: 30           # Heartbeat cadence in seconds (default: 30)
  usage_push_interval_secs: 10          # Usage push cadence in seconds (default: 10)
  usage_batch_max_records: 500          # Max records per usage push batch (default: 500)
  request_timeout_secs: 10              # Per-request outbound hub call timeout in seconds (default: 10)
  config_sync:
    enabled: false
    poll_interval_secs: 5
    state_file: "./data/control-plane/config-sync.json"
    backends:
      mode: disabled                    # disabled | overlay | authoritative
      state_file: "./data/control-plane/backend-sync.json"
      tasks: true                       # Outbound candidate-probe and redacted-export task channel
      max_concurrent_tasks: 2           # Concurrent Hub backend tasks (1-8, default: 2)

# Client-facing System One decisions: POST /v1/decisions (alias /v1/systemone).
# A name under `targets` is passed through to that System One server; any
# other name is emulated on the chat model of that name (one forced
# `emit_decision` tool call, answer marked `emulated: true`). Reloads
# immediately. See the Decisions section of api.md.
decisions:
  enabled: true                         # false answers 404 decisions_disabled
  targets:
    - name: jev                         # The `model` clients send; usable in fallback chains
      protocol: typesafe                # generic (default) | typesafe | foundry | llama_cpp
      auth: auto                        # auto (default) | bearer | api_key | entra
      endpoint: "https://api.typesafe.ai/v1/systemone"
      api_key_env: TYPESAFE_API_KEY     # Or `backend: <self-hosted backends[] name>`
      model: jev-1.14.0                 # Upstream model or Foundry deployment name
      typesafe_images: false            # Opt in to TypeSafe JSON assets and Box questions
      typesafe_long_context: false      # Opt in to TypeSafe 262k long-context header
      max_questions: 64                 # 1-256 (default: 64)
      max_state_bytes: 16384            # 256-1048576 (default: 16384)
      max_media_bytes: 4194304          # 1-16777216 decoded media bytes (default: 4 MiB)
      timeout_ms: 1500                  # 100-60000 (default: 1500)
      max_concurrent: 32                # 1-1024 (default: 32)
  emulation:
    enabled: true                       # false: non-target names answer 404
    max_state_bytes: 16384              # 256-1048576
    max_tokens: 1024                    # 16-32768

# Per-request quality judgment on control-plane usage records (needs the
# control-plane build and control_plane.enabled). A System One decision
# model judges a /v1/chat/completions reply, buffered or streamed, after it
# passes every guardrail stage; the probabilities ride on the usage record
# as basis points with the resolved judge model.
# The response, the record's accounting, and its push never wait for the
# judgment. Reloads immediately.
usage:
  quality_judgment:
    target:                             # the shared System One target block
      endpoint: "http://laya.internal:8000/v1/systemone"
    sample_rate: 1.0                    # share of eligible requests judged
    models: []                          # served model ids; empty = every model
    include_tool_results: false         # true sends the current turn's tool results (for `grounded`)
    max_request_bytes: 512              # head-and-tail caps, sized for Laya multilingual
    max_reply_bytes: 1536
    max_tool_results_bytes: 1024
    max_in_flight: 32                   # judgments in flight per router; excess is shed
  # Post-stream engine /tokenize correction (off by default): a
  # stream recorded with the router's own delta count is recounted by the
  # serving engine's tokenizer route (vllm, sglang, llamacpp, mlxcel,
  # continuumrouter) after the client stream ended, and recorded as
  # `engine_tokenize` at most timeout_ms later. Read when each stream ends.
  tokenize_correction:
    enabled: true                       # default false
    timeout_ms: 2000                    # 100-30000
    max_concurrent: 4                   # 1-256 per router; excess is shed, never queued
  # Append-only local copy of every usage batch the control-plane agent builds,
  # with its SHA-256 chain hash. Verify a file with
  # `python3 -I scripts/verify-usage-ledger.py <file>`. Reloads at the next push tick.
  ledger_export:
    path: "/var/lib/continuum-router/usage-ledger.jsonl"  # unset = no export
    max_bytes: 1073741824               # rotate past 1 GiB (minimum 1 MiB)
    rotate: true                        # false stops the export at max_bytes
    max_files: 30                       # rotated files kept (at least 1); omit to keep all
  # Ledger output tokens against engine generation counters, per backend and
  # window. Needs engine_stats.enabled: true. Reloads immediately.
  reconciliation:
    window: "5m"                        # 10s-24h; keep >= 2x engine_stats.interval
    max_drift: 0.05                     # (0, 1]
    history: 24                         # closed windows kept per backend (1-1000)

Usage ledger export¶

usage.ledger_export writes every usage batch the control-plane agent builds, one JSON line per batch with its chain hash, to an append-only, owner-only (0600) file the router never truncates. Rotated files are deleted only when max_files is set. It needs a control-plane build and control_plane.enabled: true. A value that breaks a rule in the table below is a load error, so startup, hot reload, and config validate refuse it. See Tamper evidence for the line format, the rotation anchor, and the audit procedure.

Internal streaming for buffered requests¶

proxy.internal_streaming (default false) meters buffered client requests that are cancelled mid-generation. When enabled, eligible buffered (stream absent or false) /v1/chat/completions and /v1/completions requests are dispatched to the backend with stream: true and stream_options: {include_usage: true} (plus continuous_usage_stats: true on vllm backends). The router reads the upstream SSE stream and returns a standard non-streaming JSON body to the client; if the client disconnects mid-generation, the router records a single outcome: client_cancelled entry using its count of the streamed deltas. Without internal streaming, mid-generation cancellations leave no usage record.

proxy:
  internal_streaming: true   # immediate reload

backends:
  - name: vllm-a
    type: vllm
    url: http://vllm-a:8000
    internal_streaming: false   # this backend stays buffered

A request is internally streamed only when all of these hold:

• It arrived on the client /v1/chat/completions or /v1/completions route (including its retries, fallback hops, deduplication, web-search turns, quality hops, and disaggregated or Dynamo shim dispatch). The /v1/decisions emulation, the Anthropic Messages and Responses ingresses, embeddings, rerank, images, files, synthetic probes, and every router-initiated call stay buffered.
• The backend type is vllm, sglang, tensorrt-llm, llamacpp, mlxcel, ollama, lmstudio, openai, generic, or continuumrouter, over HTTP(S).
• n is absent or 1, logprobs is absent, false, or null, top_logprobs is absent, and response_format.type is absent, text, or json_object.
• The request sets none of functions, function_call, audio, or web_search_options, and modalities does not include audio: the assembled body does not rebuild the output they ask for (function_call, audio, annotations).

The client still receives a buffered body: content, reasoning_content, refusal, tool_calls (with content: null when only tool calls were generated), the per-choice finish_reason, and the engine's usage when the engine sent one. Guardrails, the response cache, and the quality check see that body. Usage records:

• Completed stream: Records the served row; if the engine omitted a usage block, it records usage_missing with the router's token count.
• Client cancellation: Records one client_cancelled entry with the router's token count, stream: false, and http_status: 200. A cancellation after response headers arrive but before the first data event also records an attempt (client_cancelled, zero output, and estimated input).
• Failure after output starts: Records upstream_error (or timeout_mid_stream) with the partial token count and returns 502 (or 504). This outcome is final: requests that have already generated partial output are never retried or sent to a fallback hop (including disaggregated unified fallback).
• Failure before any output: Retries and falls back like a failed buffered read when no event arrived, an error event arrived, a transport error or timeout occurred after a role-only chunk, or the stream ended with neither [DONE] nor a finish_reason. Any delta field other than role (text, reasoning, tool call, or refusal) counts as output and prevents re-sending. After output has started, a stream that ends without [DONE] and without a finish_reason is treated as a mid-stream failure, whereas one that includes a finish_reason is treated as complete.
• Non-streaming upstream fallback and size limit: If a backend answers the stream request with a single application/json body, the router passes it through unchanged (logged once per backend). Internally streamed responses are capped at 16 MiB of cumulative SSE wire data, checked before parsing each read; exceeding the cap returns 502 without retry or fallback and records the usage observed so far as upstream_error.

Usage on chat streams¶

The router asks every backend for usage on a /v1/chat/completions stream (stream_options.include_usage: true) so it can meter the request, and some engines, such as a Dynamo frontend, send usage even when asked not to. The client's own include_usage decides what its stream carries, whatever the model:

• include_usage absent or false: the client receives no usage at all. The usage field is removed from every chunk, null included, and the trailing usage-only chunk (choices: [] with usage) is dropped.
• include_usage: true: the client receives usage only on the trailing usage-only chunk, the shape the OpenAI stream sends. A chunk that carries usage: null passes as it was, and a chunk that carries a non-null usage with choices has the usage removed (the running usage a per-chunk engine reports). A stream that reaches [DONE], or ends, without a usage-only chunk after such a chunk gets one from the router, carrying the last removed usage, so an engine that reports usage only on its last content chunk still delivers it once.

Everything else passes unchanged: content chunks, [DONE], keep-alives, and event: error. Metering, the response cache, and the guardrail gate read the stream before the filter, so usage records are the same either way.

• Scope: only /v1/chat/completions streams. Non-streaming responses, /v1/completions, the Anthropic Messages ingress, and the Responses API are not affected.
• Deprecated field: proxy.honor_include_usage_models is deprecated and ignored since this behavior became the default. It still parses, and listing a model changes nothing.

Per-response cost headers¶

proxy.response_cost_headers (default false) adds cost and token metadata to responses that came from a backend: as response headers on a non-streaming reply, and as HTTP trailers on a streaming one.

proxy:
  response_cost_headers: true   # immediate reload

• Routes: buffered /v1/chat/completions (including proxy.internal_streaming requests), /v1/completions, and /v1/responses requests served through the chat-completions bridge.
• Same numbers as the usage record: the token counts come from the same usage block the stats and the usage ledger record. When a quality-fallback hop replaces the reply, the headers describe the hop: its model's pricing, its tokens, and its latency.
• Missing data: a model without pricing metadata gets every header except the cost header. A response without a usage block gets only the latency header.
• Never attached: response-cache hits (no backend round trip to bill). Embeddings, rerank, images, audio, and the Anthropic Messages ingress do not carry them either.
• Cost in the usage block: independently of this toggle, a buffered /v1/chat/completions response from a model with pricing metadata carries usage.cost and usage.cost_details.upstream_inference_prompt_cost / upstream_inference_completions_cost in USD, from the same rates as the cost header (issue #2404). Each axis is rounded to a whole micro-USD, so usage.cost can differ from the header by one micro-USD. A model without pricing gets no cost fields, and a response-cache hit replays the cost stamped when the reply was stored.

The cost is computed from the router's own model catalog, not from the provider's invoice, and the toggle applies to every API key. Turn it on only when every client may see the catalog prices. Browser clients read the headers only when cors.expose_headers lists them.

Streaming responses¶

A streaming response's head leaves before its usage is metered, so a stream carries the same five fields as HTTP trailers instead of headers. The toggle is the same one; there is no separate stream setting.

• Announcement: the head carries Trailer: x-continuum-cost-microusd, x-continuum-tokens-in, x-continuum-tokens-out, x-continuum-tokens-cached, x-continuum-latency-ms.
• Values: the trailer section after the last frame carries them, with the same pricing, the same omissions as the headers, and the usage of the hop that produced the stream. x-continuum-latency-ms is measured from request start to the finalize instant.
• Negotiation: on HTTP/1.1 a client asks for the fields with TE: trailers, and hyper encodes the trailer only then. Without that header the response still carries the complete body, with no trailer fields, because they are optional metadata. HTTP/2 carries trailers without negotiation.
• Scope: the metered streaming pipelines, /v1/chat/completions, /v1/completions, /v1/responses, and their bridges. Other Server-Sent Events surfaces, the Anthropic Messages ingress and a buffered SSE reply, carry neither the Trailer header nor the fields. A stream that ends before finalizing usage (a client that left mid-stream, an upstream failure, a guardrail block) sends no trailer frame.
• Browsers: cors.expose_headers exposes response headers to browser JavaScript, but browsers cannot read trailers, so a browser client needs the usage ledger or the buffered surface for per-request stream cost.

Hedged requests¶

proxy.hedge_delay_ms (unset by default, 0 to 60000) races a slow dispatch against a second backend. When a buffered /v1/chat/completions or /v1/completions dispatch has not answered this many milliseconds after its first backend was selected, the router sends the identical request to a second admissible backend serving the same model, adopts whichever response arrives first, and cancels the other. 0 sends both at once.

proxy:
  hedge_delay_ms: 500   # immediate reload; unset disables hedging

• Never hedged: client streams (including a legacy /v1/completions request with stream: true), fallback-chain hops, embeddings, rerank, and image generation. A backend with replay: disabled is never hedged and never chosen as the second target, because a cancelled attempt has already received the body.
• Second target: chosen through the same model, visibility, and allow-list pipeline as the first, excluding the first backend; its circuit breaker and rate-limit admission are checked when it dispatches, and a refusal leaves the first attempt running alone.
• Accounting: both attempts count against their backends' circuit breakers and dispatch admission. The cancelled attempt is recorded as its own usage row with outcome: hedge_cancelled, and the winner's row is the request's own; a request that sent an x-backend preference sees the winner in x-served-backend.

Model variant suffixes¶

proxy.variant_suffixes.enabled (default false) adds OpenRouter-style variant suffixes to /v1/chat/completions (issue #2405). With the switch on, a model name that does not resolve exactly may end in one recognized suffix; the router splits it at the last : and resolves the base id. The full requested name is resolved first, so a name a backend serves literally, such as an Ollama name:tag, keeps working unchanged, and every other ingress keeps its existing behavior.

proxy:
  variant_suffixes:
    enabled: true   # default false; immediate reload

• Never a silent substitution: :nitro and :floor never fail. When no candidate carries fresh throughput or pricing data, normal selection serves the request and the response carries x-continuum-model-variant-applied: false next to x-continuum-model-variant, which names the variant word.
• Refusals: :thinking on a model whose metadata declares no thinking support, :online when no web_search executor is available (disabled, commercial backend, or no configuration), and :online on a streaming request answer 404 with error.type: "invalid_request_error" and error.code: "unsupported_model_variant", naming the suffix. An unknown suffix word is an unknown model, exactly as before.
• Identity and accounting: per-key allowed_models and the per-model rate limit key on the base id, which is also the id dispatched upstream. The requested name (base plus suffix) is echoed in the response model, the base id is what the backend sees, and a fallback-chain hop never inherits the variant.
• Observability: every request answered for a suffixed name counts in continuum_model_variant_requests_total{variant, applied}, where applied is false when no ranking data was available or a per-backend web_search override declined injection.

Middle-out message transform¶

proxy.middle_out fits an over-window /v1/chat/completions request instead of returning the backend's context-length error. The operator switch and the client opt-in are both required; without them the request is forwarded unchanged.

proxy:
  middle_out:
    enabled: false                 # immediate reload; false by default
    default_output_reserve: 1024   # 0 to 1000000; used when the request sets no output limit

• Client opt-in: a top-level "transforms": ["middle-out"] array in the request body, matched ASCII case-insensitively. A transforms value that is not an array is ignored.
• Fit rule: the shared input estimate must exceed the model's context window minus the output budget. The budget is max_completion_tokens, else max_tokens, else default_output_reserve, and the window comes from the same limit snapshot the context-window admission check uses.
• What is dropped: whole messages from the middle, oldest first. The leading system/developer run and the current turn (the last user message and every later message) never drop, and an assistant tool call drops together with its tool results.
• Signal: a fitted response carries x-continuum-middle-out: dropped=<n>;estimated_input_tokens=<n>;target_input_tokens=<n>;context_window=<n> on buffered and streaming responses alike, and the spent middle-out entry is removed from transforms before dispatch.
• Never fitted: when the kept messages alone exceed the target, the payload is forwarded unchanged and the request takes its ordinary path, the context-window fallback when configured and the backend's error otherwise. A fitted request passes the admission check, so it never also takes that fallback hop.

Usage reconciliation¶

usage.reconciliation compares the output tokens recorded in the usage ledger against the delta of the engine's cumulative generation counter scraped by engine_stats per backend and per window. Results are exposed at GET /admin/usage/reconciliation and through the usage_reconciliation_* metrics.

• Supported backend types: Only vllm, sglang, llamacpp, and mlxcel backends report the engine counter (llamacpp requires --metrics). All other backend types report no_engine_data, and config validate warns when they are included. Set reconciliation: false on a backend entry to exclude it (for example, when an engine is shared with other routers or clients).
• Build and validation: The block is always compiled and does not require the control-plane feature. Values outside the ranges in the table below fail configuration loading. See Reconciliation against engine counters for window rules.

Recording responses without a usage block¶

backends[].record_usage_missing (default false; alias record_missing_usage, immediate reload) opts a backend into emitting a zero-token usage record when a non-streaming 2xx response on /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/rerank, /v1/responses, or /anthropic/v1/messages carries no usable usage block (for example a self-hosted embedding or rerank server, or an OpenAI-compatible proxy that omits usage on buffered responses).

backends:
  - name: "self-hosted-embeddings"
    type: vllm
    url: "http://embeddings:8000"
    record_usage_missing: true   # default false; immediate reload

When false (default), a non-streaming 2xx response with no usage block emits no Hub usage row. When true, responses that already carry a valid usage block or were finalized through proxy.internal_streaming still emit their single normal usage record, and 4xx/5xx or transport failures are unchanged. Because all token counts on the synthesized row are 0, it adds nothing to priced token cost, key TPM windows, monthly token deltas, or cost-center token budgets.

Minimal Configuration¶

# Minimal configuration - other settings will use defaults
server:
  bind_address: "0.0.0.0:8080"

backends:
  - name: "ollama"
    url: "http://localhost:11434"
  - name: "lm-studio"
    url: "http://localhost:1234"

health_checks:
  interval: "30s"
  timeout: "5s"
  unhealthy_threshold: 3
  healthy_threshold: 2
  endpoint: "/health"

logging:
  level: "info"
  format: "json"

Dynamo event broker listener¶

dynamo.event_bus runs the router side of the in-cluster event broker tunnel: a dedicated TLS listener that an in-cluster continuum-event-broker dials out to, allowing a router outside the Kubernetes cluster to reach worker event publishers it cannot dial directly.

Active operation requires the dynamo-event-bus Cargo feature (built on top of dynamo; included in official release binaries, but not in default/full). The configuration block is always compiled and validated, and a binary built without the feature logs at startup that the broker listener is inactive. It operates independently of dynamo.enabled. See Dynamo Worker Discovery for protocol details, and docs/shared/llm-skills/config-templates/dynamo-event-bus.yaml for a runnable template.

Nothing in the block needs a restart. On a hot reload:

• enabled starts the listener, or sends every session drain {reason: shutdown, deadline_ms: 2000}, ends it with a terminal draining error after 2s, and stops the listener.
• listen restarts the listener on the new address.
• tls.* re-reads the PEM files and uses them for new connections; established connections keep theirs. A client_ca change also closes every certificate-authenticated session with credential_revoked, and a connection whose certificate was verified under the previous client_ca can no longer authenticate with it (a new session on it is 401 credential_revoked). Only a change to a path is seen as a change: replacing the contents of a cert, key, or CA file in place is not picked up until a path changes or the router restarts, which matters when revoking a CA.
• Removing a broker or changing its token_hash closes that broker's live session with credential_revoked, and the removed credential is then answered 401 credential_revoked rather than 401 unauthenticated.
• Narrowing a broker's namespaces so a live session's declared namespaces are no longer covered closes that session with forbidden. Narrowing a kv_listener entry's backends, or changing an entry's role, closes a live session that no longer matches with forbidden.
• The stream and buffer limits drain and restart the listener to replace its HTTP/2 windows. max_sessions and both timers apply to new sessions.

Broker token hashes are redacted in debug output and masked configuration exports. Importing a masked export restores each broker's existing hash by its id, including after reordering.

A TLS connection has to earn its slot under the connection cap: it is closed (auth_timeout in continuum_dynamo_event_bus_connections_refused_total) if no request is accepted within 10s of the TCP accept, and it is closed 30s after its last accepted request ended. Refused requests neither count nor reset that timer. These two deadlines are fixed and not configurable.

A listener that fails to start (a missing TLS file, a port in use) is logged as event bus listener failed to start; retrying and retried every 30s until it succeeds or the section changes.

Subscribing through a broker¶

dynamo.load.transport decides how the kv_metrics load subscriptions and the kv-events KV subscriptions reach their ZMQ publishers:

When dynamo.load.transport is set to broker, subscription lifecycle and observability work as follows:

• Pending state and startup jitter: A subscription not covered by any live broker session remains pending (nothing is dialed, and its worker is treated as having no fresh load sample). Once a covering broker session is welcomed, the subscription starts after a random delay of up to 500ms so a newly connected broker does not absorb all pending subscriptions in a single burst.
• Rendezvous placement and failover: When multiple sessions cover a namespace, each subscription is assigned to one broker via a stable rendezvous hash of the publisher and broker ids. Subscriptions spread evenly across covering brokers and return to the same broker on reconnect. If a broker session ends, its subscriptions reconnect through the next-ranked covering broker after the minimum backoff floor (at most 500ms, without escalation), or return to pending if no covering broker remains. kv-events ranks preserve their cursors across broker failover; a kv-events subscription that remains disconnected for 60s has its ranks cleared from the KV index.
• Build requirements: broker requires the dynamo-event-bus build feature and an enabled dynamo.event_bus block. A binary compiled without the feature logs a startup warning and keeps subscriptions pending rather than falling back to direct dialing.
• Hot reload: Changing dynamo.load.transport takes effect immediately. Both subscription supervisors restart their subscriptions over the new transport without disturbing worker discovery, the registry, the load store, or the KV index.
• Observability (GET /admin/dynamo/workers and metrics): The admin endpoint lists every running subscription under subscriptions with its family, publisher_id, namespace, component, state (pending, connecting, connected, backoff), transport, and carrying broker id (the publisher endpoint is never exposed). Subscription gauges carry a transport label, and continuum_dynamo_broker_subscriptions{family, broker} counts connected subscriptions per broker. Under broker, the endpoint also includes broker_coverage (showing per-namespace coverage, covering broker ids, and pending subscriptions per family), mirrored by the continuum_dynamo_broker_namespace_sessions{namespace} (0 when uncovered) and continuum_dynamo_broker_pending_subscriptions{family, namespace} metrics.

Environment Variables¶

Environment variables enter the configuration through two mechanisms:

1. Direct overrides: a small set of CONTINUUM_* variables that override their config-file counterparts at startup.

2. General ${VAR} interpolation: every string-valued YAML/TOML field is interpolated when the file is loaded, not just secrets. logging.level: "${LOG_LEVEL}", files.storage_path: "/data/${TENANT}", timeout strings, URLs, and paths all work.

   • ${VAR} expands to the value of VAR.
   • ${VAR:-default} expands to VAR if set, otherwise the literal default (which may be empty).
   • $$ is an escape producing a literal $, so $${VAR} yields the literal text ${VAR}.
   • Variable names must match [A-Za-z_][A-Za-z0-9_]*; a ${...} with an invalid name is left literal. Only string scalars are interpolated, so a ${VAR} where a number or boolean is expected fails typed parsing.

   At startup and hot-reload, an unset ${VAR} with no default aborts loading with an error naming the config path. continuum-router config validate instead reports the unset reference as a warning and keeps the placeholder, so a template can be validated without production secrets present.

Direct Overrides¶

Native Backend API Keys¶

Native backends fall back to these variables when no api_key is set in the config:

API Key Management¶

Model Metadata Download Token¶

continuum-router metadata download reads a GitHub token from these variables, in order, and uses the first non-empty one. See Model metadata download for the full rules.

The token is required while the repository is private, is sent only to raw.githubusercontent.com and api.github.com, and is never logged or printed.

Example Environment Configuration¶

# Direct overrides
export CONTINUUM_BIND_ADDRESS="0.0.0.0:9000"
export CONTINUUM_BACKEND_URLS="http://localhost:11434,http://localhost:1234"
export CONTINUUM_LOG_LEVEL="debug"

# Referenced from config.yaml via ${...} substitution
export CONTINUUM_OPENAI_API_KEY="sk-..."
export ADMIN_TOKEN="my-admin-token"

# Start the router
continuum-router

All other settings (health checks, timeouts, retry, caching, and the rest of the Files API configuration beyond files.auth) are configured through the configuration file.

Command Line Arguments¶

Command-line arguments provide the highest priority configuration method and are useful for testing and temporary overrides.

Core Options¶

continuum-router --help

Backend Configuration¶

Server Configuration¶

Load Balancing¶

Health Check Configuration¶

Configuration Utilities¶

Container Health Check¶

Subcommands¶

Model metadata download¶

model-metadata.yaml and the vendor/developer fragments in model-metadata.d/ carry router-wide pricing, context windows, capabilities, thinking patterns, and /v1/models response defaults. Release archives, Debian packages, and container images include both the compatibility base catalog and the split fragments. metadata download can refresh them between binary releases.

export CONTINUUM_GITHUB_TOKEN="$(gh auth token)"    # required while the repository is private

continuum-router metadata download                  # fetch main, install at the resolved path
continuum-router metadata download --check          # report staleness only, write nothing
continuum-router metadata download --ref v1.15.5    # pin to a release tag
continuum-router metadata download --output ./model-metadata.yaml
continuum-router metadata update --all               # base plus configured drop-in paths
continuum-router metadata update --dir ./model-metadata.d

Every upstream-managed file declares a top-level HTTPS source_url. A single-file update uses that origin when the destination already exists and --url is absent; the built-in repository URL is used only for a new file or a legacy file without source_url. --ref rewrites the ref portion of origins below the continuum-router raw-content repository and leaves third-party origins unchanged.

--all scans the resolved base file and the complete configured drop-in search path. Each --dir <DIR> adds an explicit drop-in directory and may be repeated. Files without source_url are local operator files: bulk mode reports them under skipped_local and never writes them. --check, --force, --no-backup, --ref, and --json apply to every tracked file; --check exits 2 when any one of them differs. --url and --output are single-file options and therefore conflict with bulk mode.

Bulk planning uses the same bounds as runtime assembly: at most 64 base/drop-in files and at most 8 MiB of catalog data in total. Source-less local files consume that byte budget, and downloaded replacements share the remainder. Every tracked replacement is fetched and schema-validated before the first file is installed, so a failed or oversized later source cannot leave an earlier file updated while the rest of the catalog remains stale. Single-file mode also rejects a response larger than 8 MiB.

Authentication¶

The continuum-router repository is private, so an unauthenticated request to the default source returns 404 for every ref, including refs that certainly exist. Supply a GitHub token with read access through one of these variables, checked in order with the first non-empty value winning:

Surrounding whitespace is trimmed and an all-whitespace value counts as unset, so a stray export GITHUB_TOKEN= does not shadow a real GH_TOKEN.

There is no --token flag on purpose: a secret passed as a command-line argument lands in shell history and in ps output for every user on the machine.

The token is attached only when each request host is exactly raw.githubusercontent.com or api.github.com. A --url or file-supplied source_url pointing at a vendor mirror, or at a lookalike host such as raw.githubusercontent.com.evil.test, never receives it. Redirects must remain HTTPS and are capped at ten hops. Redirect requests do not generate a Referer header, preventing a signed source URL from exposing its query credential to the next origin. The token is never logged or printed; the report names the variable it came from, not its value.

A 404 tells you which case you are in. Without a token, the error names the three variables and says the repository may be private. With a token, it says the ref or path may not exist or the token may lack access.

A --url may carry basic-auth credentials for a private mirror (https://ci:token@mirror.internal/model-metadata.yaml). They are sent as an Authorization: Basic header, while every displayed source (the human report, the --json source_url field, layer reports, and error messages) removes userinfo, the entire query, and the fragment. Transport errors also discard reqwest's original request URL. Fetching still uses the unmodified URL, but passwords, signed-query credentials, and fragment data do not reach CI logs or archived reports.

Flags¶

The global -c/--config and --model-metadata are reused for destination resolution.

Destination resolution¶

The destination is resolved in this order and always printed, on failure as well as on success:

1. --output <FILE>
2. the global --model-metadata <FILE>, tilde-expanded
3. model_metadata_file from the loaded config, tilde-expanded
4. model-metadata.yaml next to the discovered config file
5. ~/.config/continuum-router/model-metadata.yaml

--model-metadata sits above the config field because that is what it does for the router itself: it overrides model_metadata_file. So continuum-router --model-metadata /etc/cr/meta.yaml metadata download installs the file exactly where that router instance will read it.

Missing parent directories are created.

Safety behavior¶

• The download is parsed and validated against the metadata schema before anything on disk is touched. A malformed or schema-invalid response exits non-zero and leaves the existing file byte-for-byte unchanged.
• Replacement is atomic through a temporary file in the destination directory, and an existing file's mode is preserved (a new file gets 0644).
• The previous file is kept as <file>.bak unless --no-backup is given.
• Content is compared by SHA-256, so an unchanged file is skipped without a write unless --force is given.
• Only HTTPS sources are accepted, and redirects that leave HTTPS are refused.
• Responses larger than 8 MiB are rejected.

Exit codes¶

The 2 case lets cron and CI branch on staleness without treating it as an error.

Effect on a running router¶

Updating the file does not hot-reload a running router. The config watcher watches only the config file path, and both model_metadata_file and model_metadata_dirs are marked requires_restart in the admin config schema, so restart the router or trigger a config reload for the new metadata to take effect. The command prints this reminder on every run.

Layered model metadata¶

model_metadata_file is layer zero, not the whole story. Metadata is assembled from a layered search path. The shipped catalog is split into vendor/developer fragments with low numeric prefixes, while operator files normally use a later prefix such as 90-; bulk updates touch only files that opt in with source_url.

Search path¶

Lowest precedence first, so the more specific location wins by being applied last:

1. model_metadata_file (the base file, tilde-expanded)
2. /etc/continuum-router/model-metadata.d/
3. ~/.config/continuum-router/model-metadata.d/
4. ./model-metadata.d/
5. every directory listed in model_metadata_dirs, in the order given

model_metadata_dirs supplements the conventional locations rather than replacing them, and its entries are used verbatim: the model-metadata.d name is not appended, so a Kubernetes ConfigMap mounted at /etc/router/metadata/ is listed as exactly that path. The global --model-metadata flag keeps its meaning: it sets the base file, it does not disable drop-ins.

A typical layout:

/etc/continuum-router/model-metadata.yaml        # vendor baseline, safe to overwrite
/etc/continuum-router/model-metadata.d/
    10-alibaba.yaml                              # shipped, source_url-bearing fragment
    23-openai.yaml                               # shipped, source_url-bearing fragment
    50-negotiated-pricing.yaml                   # local: no source_url, never bulk-updated
    90-local-overrides.yaml

File selection¶

Within a directory:

• only regular files ending in .yaml or .yml are read, matched case-insensitively
• files apply in lexicographic filename order, which is what makes the 10-/50-/90- numeric prefix convention work
• subdirectories are never recursed into
• dotfiles, *~, and *.swp are skipped, so editor backups and vim swap files are ignored
• a symlink is resolved but must still land on a regular file
• a missing directory is not an error

Merge rules¶

Layers merge on the parsed YAML, before deserialization:

• mapping into mapping: recursive key-wise merge, so keys the higher layer omits inherit from the lower one
• sequence: replaced wholesale, never unioned
• scalar (including an explicit null): replaced
• models entries are keyed by id: an entry whose id already exists merges into the existing entry, and a new id is appended
• response_defaults is a mapping, so it merges per field

Merging before deserialization is what makes a narrow override possible. An override that never mentions responses_only cannot clear an inherited true, and a field added to the metadata schema later merges correctly with no new code.

Sequences replace rather than union deliberately. aliases and capabilities are sequences, and union semantics would make removing an inherited entry impossible.

Given a vendor baseline:

models:
  - id: gpt-5
    aliases: [gpt-5-latest]
    metadata:
      display_name: GPT-5
      capabilities: [chat, reasoning, vision]
      pricing: { input_tokens: 1.25, output_tokens: 10.0 }

and model-metadata.d/50-negotiated-pricing.yaml:

models:
  - id: gpt-5
    metadata:
      pricing: { input_tokens: 0.90, output_tokens: 7.20 }

the effective result keeps display_name, capabilities, and aliases tracking upstream while pricing uses the negotiated rates. The operator maintains four lines instead of a full copy that rots.

Retired models¶

Models that have been shut down across all serving surfaces (the provider's native API, Azure OpenAI, and Vertex AI) are moved out of the base catalog into model-metadata-retired.yaml at the repository root. This keeps /v1/models metadata and smart-routing profile inference focused on models that requests can still reach.

The router never loads model-metadata-retired.yaml automatically. The file uses the same schema as model-metadata.yaml and records where and when each model was retired. If your deployment still serves one of those model ids (for example, behind a self-hosted OpenAI-compatible gateway), copy the file or the relevant entries into a model-metadata.d/ directory to restore its metadata. Any model still active on at least one surface remains in the base catalog, and no id or alias appears in both files.

Precedence outside the metadata layer¶

Unchanged: backend model_configs still win over the merged metadata, which still wins over the built-in OpenAI registry. Layering changes only how the metadata layer itself is assembled.

Model artifact identity¶

A model id names what a client asked for, not which weights answered. Two keys let the operator say which artifact a model is served from, so usage records can tell apart two backends that serve one id from different weights (another quantization, another revision, a re-exported checkpoint):

# model-metadata.d/10-private-models.yaml
models:
  - id: qwen3.8-27b
    metadata:
      artifact:
        digest: sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
        registry_version: v2.1-fp8
        revision: 4c2e9a1
        source: hf://Qwen/Qwen3.8-27B-FP8
      draft_model:
        name: qwen3.8-0.6b
        artifact:
          digest: blake3:5e2b0a9c1f7d3e8b4a6c0d2f9e1b7a3c5d8f0e2a4c6b8d0f1e3a5c7b9d0f2e4a

The identity resolves per backend and model, one key at a time: a backend's own model_configs entry wins when it sets the key, otherwise the merged metadata layer does. A backend that sets artifact replaces the shared artifact whole (its sub-fields are never merged with the shared one), and draft_model resolves on its own, so a backend can override the digest and still inherit the shared draft. The built-in OpenAI registry is never consulted, since a cloud model has no artifact an operator can vouch for.

Every usage record that reached a backend carries the resolved model_digest and draft_model (see Artifact identity on usage records). draft_model names the draft and counts nothing: engines report accepted tokens only, which belong to the target model.

How a malformed value is handled depends on where it is defined:

• In backends[].model_configs: Treated as a configuration load error. config validate reports the exact path (for example, backends[1].model_configs[0].metadata.artifact.digest) and names the model.
• In model-metadata.yaml or a drop-in: Follows the same warn-only rule as an out-of-range price. The model is still served, a single warning names the model, and the malformed artifact (or draft_model) block is dropped so it never reaches /v1/models, /admin/models, or usage records. metadata show --resolved still prints the raw value and lists the violation under identity_warnings, while metadata download refuses any catalog containing a malformed identity.

config validate also warns, without failing, when a vllm, sglang, or tensorrt-llm backend's entry names a draft_model without a digest for the target or for the draft, and once per model id that model_configs declares with different digests on different backends (expected when the backends deliberately serve different builds).

Failure handling and limits¶

A drop-in file that fails to read, parse, or validate is skipped with a warning while the rest of the search path still loads, preventing a single malformed operator file from taking down a running router. A malformed base model_metadata_file, however, is a fatal load error.

Schema failures are attributed to the specific layer that introduced them: a layer is blamed only if the merged state was valid immediately before that layer was applied. If the base file itself violates the schema, the error points to the base file rather than to drop-ins layered on top of it. metadata show reports this under Schema failure introduced by.

At most 64 files totalling at most 8 MiB are assembled. Exceeding either cap is an error, not a truncation: applying a partial override set silently would be worse than refusing to assemble.

A fatal assembly failure during a hot reload aborts the entire reload, not just the metadata layer. The router logs an error naming the metadata assembly as the cause and continues serving with the previously loaded configuration and model metadata, preventing a partial reload from silently reverting pricing, context windows, capabilities, and /v1/models defaults to backend-reported values. Note that adding a 65^th file or pushing the total size past 8 MiB in a model-metadata.d/ directory will trigger this assembly failure on the next reload.

Inspecting the result¶

Skipping is invisible to clients, so metadata show is the supported way to see it:

continuum-router metadata show              # the raw merge, with the layer report
continuum-router metadata show --resolved   # the effective typed document the router loads
continuum-router metadata show --json       # the same report as a machine-readable object

• Common output: Both modes print the search path, the applied layers in application order (including each redacted source_url, which model ids each later layer overrode, and from which file), and any skipped files with their failure reasons. Because the report header is emitted as YAML comments, metadata show > merged.yaml produces a valid YAML file that the router can load directly.
• Default vs. --resolved mode: Without --resolved, the command prints the raw YAML merge verbatim, including unrecognized keys so typos remain visible. With --resolved, the merged document is deserialized, response_defaults is validated and sanitized, and the typed structure is re-serialized: artifact digests appear in canonical lowercase, and any malformed artifact identities dropped at load time are listed under # Artifact identity warnings (identity_warnings in --json) while their raw values are still shown.

metadata show exits 0 whenever the search path could be assembled, including when files were skipped, and 1 on a fatal failure such as an unreadable base file or an exceeded cap.

If the discovered or explicitly named config file could not be found or could not be loaded, metadata show says so instead of silently describing a search path that omits that config's model_metadata_file and model_metadata_dirs: a # WARNING: header line in the human report, and a config_warning field alongside --json.

Relationship to metadata download¶

Without bulk flags, metadata download updates one resolved file and prefers that file's source_url. With --all or --dir, it updates every source-bearing base or drop-in file and explicitly skips files without source_url. This per-file opt-in is the safety boundary: shipped vendor fragments are disposable and reproducible, while operator files remain untouched.

Drop-in changes do not hot-reload either. model_metadata_dirs is requires_restart in the admin config schema, same as model_metadata_file.

Status dashboard¶

continuum-router status reads the Admin API of a router that is already running and prints a dashboard. It does not load the local configuration, so it works from any host that can reach the router.

The dashboard has three blocks:

• Backends: health status, circuit-breaker state, request and failure counts, and average latency for each backend.
• Models: the top models by requests, with a share bar, token totals, tokens per second, and average latency.
• Overall: request count and success rate, prompt and completion tokens, latency percentiles, and the circuit-breaker summary.

The command fetches GET /version, GET /admin/stats, and GET /admin/circuit/all, so the router must be built with the admin feature (included in default and official release builds).

Authentication follows admin.auth.method:

• api_key (the default): pass an API key that has the scope named by admin.auth.required_scope (default admin).
• bearer: pass the configured admin.auth.bearer_token.
• none or ip_whitelist: no token is needed.
• basic: not supported by this command.

Prefer CONTINUUM_ADMIN_TOKEN over --token. Other local users can read command-line arguments from the process list, while the environment of a process is private to its owner. The token is never printed, and the client does not follow redirects, so the token cannot be forwarded to another origin.

The command exits 0 when a snapshot was fetched and rendered, and 1 when the router cannot be reached or answers with an error. A 401 or 403 answer adds a hint about the token. In --watch mode a failed fetch redraws the screen with the error and keeps polling, so a stalled router stays visible.

Colors are turned off when stdout is not a terminal or NO_COLOR is set. Model and backend names are printed with control characters replaced, so a name reported by an upstream backend cannot inject terminal escape sequences.

Usage report¶

continuum-router usage reads the Admin API of a running router and prints a token usage report with one row per model, backend, API key, or user. Like status, it does not load the local configuration.

Rows are ordered by tokens (requests for backends), largest first, and --top keeps the first N. --series appends the global daily token series from GET /admin/stats/series as one bar per UTC day, split into prompt (█) and completion (▓) tokens. The router serves at most its configured series retention, whatever --series-days asks for.

Authentication, the exit codes, and the control-character handling are the same as for status. A router that does not serve one of the endpoints (an older version, or one built without the admin feature) makes the command exit 1 rather than print an empty report. An endpoint that answers with no rows prints no usage recorded.

Example CLI Usage¶

# Use config file with overrides
continuum-router --config config.yaml --bind "0.0.0.0:9000"

# Override backends temporarily
continuum-router --config config.yaml --backends "http://localhost:11434"

# Use custom model metadata file
continuum-router --config config.yaml --model-metadata /path/to/custom-metadata.yaml

# Use model metadata with tilde expansion
continuum-router --model-metadata ~/configs/model-metadata.yaml

# Adjust health check settings for testing
continuum-router --config config.yaml --health-check-interval 10

# Generate sample configuration
continuum-router --generate-config > my-config.yaml

# Validate it and generate a focused template
continuum-router config validate my-config.yaml
continuum-router config generate --template production-ha --output production.yaml

# Inspect or compare configurations (`--resolved` output may contain secrets)
continuum-router config show my-config.yaml
continuum-router config diff my-config.yaml production.yaml

# Migrate an outdated config file (preview first, then apply)
continuum-router --migrate-config-file config.yaml --dry-run
continuum-router --migrate-config-file config.yaml

# Docker HEALTHCHECK probe
continuum-router --health-check --health-check-url http://localhost:8080/health

# OAuth device-flow login for a ChatGPT/Codex backend
continuum-router --config config.yaml auth login --backend chatgpt

# Register the configuration assistant with an MCP client
claude mcp add continuum-router-config -- continuum-router mcp-serve

# Refresh model metadata, or check whether the local copy is stale (exit 2 = update available)
export CONTINUUM_GITHUB_TOKEN="$(gh auth token)"
continuum-router metadata download
continuum-router metadata download --check --json

# Install at the path this router instance reads
continuum-router --model-metadata /etc/continuum-router/model-metadata.yaml metadata download

# Inspect the metadata actually in effect after model-metadata.d layering
continuum-router metadata show --resolved
continuum-router metadata show --json

# Dashboard of a running router; keep the admin token out of the process list
export CONTINUUM_ADMIN_TOKEN="<admin-scoped API key>"
continuum-router status --url http://router.internal:8080
continuum-router status --watch --interval 2 --window 1h
continuum-router usage --by api-keys --top 20
continuum-router usage --window 24h --series --series-days 14