Advanced Configuration¶

Global Prompts¶

Global prompts allow you to inject system prompts into all requests, providing centralized policy management for security, compliance, and behavioral guidelines. Prompts can be defined inline or loaded from external Markdown files.

Basic Configuration¶

global_prompts:
  # Inline default prompt
  default: |
    You must follow company security policies.
    Never reveal internal system details.
    Be helpful and professional.

  # Merge strategy: prepend (default), append, or replace
  merge_strategy: prepend

  # Custom separator between global and user prompts
  separator: "\n\n---\n\n"

External Prompt Files¶

For complex prompts, you can load content from external Markdown files. This provides: - Better editing experience with syntax highlighting - Version control without config file noise - Hot-reload support for prompt updates

global_prompts:
  # Directory containing prompt files (relative to config directory)
  prompts_dir: "./prompts"

  # Load default prompt from file
  default_file: "system.md"

  # Backend-specific prompts from files
  backends:
    anthropic:
      prompt_file: "anthropic-system.md"
    openai:
      prompt_file: "openai-system.md"

  # Model-specific prompts from files
  models:
    gpt-5.6-sol:
      prompt_file: "gpt5-6-sol-system.md"
    claude-opus-5:
      prompt_file: "claude-opus-system.md"

  merge_strategy: prepend

Prompt Resolution Priority¶

When determining which prompt to use for a request:

1. Model-specific prompt (highest priority) - global_prompts.models.<model-id>
2. Backend-specific prompt - global_prompts.backends.<backend-name>
3. Default prompt - global_prompts.default or global_prompts.default_file

For each level, if both prompt (inline) and prompt_file are specified, prompt_file takes precedence.

Merge Strategies¶

REST API Management¶

Prompt files can be managed at runtime via the Admin API:

# List all prompts
curl http://localhost:8080/admin/config/prompts

# Get specific prompt file
curl http://localhost:8080/admin/config/prompts/prompts/system.md

# Update prompt file
curl -X PUT http://localhost:8080/admin/config/prompts/prompts/system.md \
  -H "Content-Type: application/json" \
  -d '{"content": "# Updated System Prompt\n\nNew content here."}'

# Reload all prompt files from disk
curl -X POST http://localhost:8080/admin/config/prompts/reload

See Admin REST API Reference for complete API documentation.

Security Considerations¶

• Path Traversal Protection: All file paths are validated to prevent directory traversal attacks
• File Size Limits: Individual files limited to 1MB, total cache limited to 50MB
• Relative Paths Only: Prompt files must be within the configured prompts_dir or config directory
• Sandboxed Access: Files outside the allowed directory are rejected

Hot Reload¶

Global prompts support immediate hot-reload. Changes to prompt configuration or files take effect on the next request without server restart.

Model Metadata¶

Continuum Router supports rich model metadata to provide detailed information about model capabilities, pricing, and limits. This metadata is returned in /v1/models API responses and can be used by clients to make informed model selection decisions.

Metadata Sources¶

Model metadata can be configured in three ways (in priority order):

1. Backend-specific model_configs (highest priority)
2. External metadata file (model-metadata.yaml)
3. No metadata (models work without metadata)

External Metadata File¶

Create a model-metadata.yaml file:

models:
  - id: "gpt-5.6-sol"
    aliases:                    # Alternative IDs that share this metadata
      - "gpt-5.6"
    metadata:
      display_name: "GPT-5.6 Sol"
      summary: "Frontier GPT-5.6 model for complex professional work"
      capabilities: ["chat", "vision", "code", "reasoning", "tool"]
      knowledge_cutoff: "2026-02"
      pricing:
        input_tokens: 5.0    # USD per 1M tokens
        output_tokens: 30.0  # USD per 1M tokens
      limits:
        context_window: 1050000
        max_output: 128000

  - id: "llama-3-70b"
    aliases:                    # Different quantizations of the same model
      - "llama-3-70b-instruct"
      - "llama-3-70b-chat"
      - "llama-3-70b-q4"
      - "llama-3-70b-q8"
    metadata:
      display_name: "Llama 3 70B"
      summary: "Open-source model with strong performance"
      capabilities: ["text", "code"]
      knowledge_cutoff: "2023-12"
      pricing:
        input_tokens: 1.0    # USD per 1M tokens
        output_tokens: 2.0   # USD per 1M tokens
      limits:
        context_window: 8192
        max_output: 2048

Image Model Pricing¶

Image models price on axes that per-token rates cannot express, so pricing accepts three further optional keys:

  # Billed per generated image: flat by size, flat by named tier, or
  # size then quality. All three shapes are supported.
  - id: "dall-e-2"
    metadata:
      pricing:
        input_tokens: 0        # literal, not missing data
        output_tokens: 0
        per_image:             # USD per generated image
          1024x1024: 0.02
          512x512: 0.018

  - id: "dall-e-3"
    metadata:
      pricing:
        input_tokens: 0
        output_tokens: 0
        per_image:
          1024x1024: { standard: 0.04, hd: 0.08 }
      limits:
        context_window: 0            # 0 = not applicable (no token context)
        max_output: 0
        max_prompt_length: 4000      # characters, not tokens
        supported_sizes: ["1024x1024", "1792x1024", "1024x1792"]
        max_n: 1                     # max images per request
        supported_qualities: ["standard", "hd"]
        supported_styles: ["vivid", "natural"]

  # Billed per token, with image input at its own rate alongside text input.
  - id: "gpt-image-2"
    metadata:
      pricing:
        input_tokens: 5.0
        output_tokens: 30.0
        image_input_tokens: 8.0          # USD per 1M image input tokens
        cached_image_input_tokens: 2.0   # USD per 1M cached image input tokens

image_input_tokens and cached_image_input_tokens are rates in the same per-1M unit as input_tokens, and they add to the text rates rather than replacing them. Note that cached_image_input_tokens is a price, unlike cached_input_discount, which is a ratio in 0.0-1.0.

per_image uses a different unit again: USD per generated image. A model that carries it reports input_tokens: 0 and output_tokens: 0, and those zeros are deliberate. The presence of per_image is what tells a client that token pricing is not that model's axis.

All three keys are optional and additive, so a catalog entry that omits them behaves exactly as before. They are served on /v1/models/extended and /v1/models/{model}; see Pricing Object Fields.

Image models describe their request surface with optional limits keys as well, shown on the dall-e-3 entry above: max_prompt_length (characters, not tokens), supported_sizes, max_n, supported_qualities, supported_output_formats, supported_styles, and supports_streaming. All of them are informational for API consumers; the router enforces image request limits per model family in code and does not read these values. context_window: 0 / max_output: 0 mean "not applicable" for models that are not token-billed, the same literal-zero convention the pricing block uses. Like the pricing keys, every limits key beyond the two token bounds is optional and additive, and an unknown key in a metadata file is ignored with a load-time warning that names its dotted path. dall-e-3's supported_qualities and supported_styles were previously spelled qualities and styles in the catalog; a drop-in that still uses the old names is now named by that same unknown-key warning instead of being silently dropped.

Both pricing and limits values are range-checked (for example context_window and max_output accept 0 to 10,000,000). On the router's own runtime load path a violation is a warning naming the model and the field, and the model keeps serving the value as written. metadata download runs the same checks but treats a violation as fatal instead, rejecting a downloaded file with an out-of-range value before anything on disk changes, because a bad download must never replace a working file.

Reference it in your config:

model_metadata_file: "model-metadata.yaml"

model_metadata_file is only the base layer. The router also assembles metadata from model-metadata.d/ drop-in directories and a model_metadata_dirs config key, so an operator's overrides do not have to live in the file metadata download overwrites. See Layered model metadata for the search order, the merge rules, and metadata show.

Thinking Pattern Configuration¶

Some models output reasoning/thinking content in non-standard ways. The router supports configuring thinking patterns per model to properly transform streaming responses.

Pattern Types:

Configuration Example:

models:
  - id: nemotron-3-nano
    metadata:
      display_name: "Nemotron 3 Nano"
      capabilities: ["chat", "reasoning"]
      # Thinking pattern configuration
      thinking:
        pattern: unterminated_start
        end_marker: "</think>"
        assume_reasoning_first: true

Thinking Pattern Fields:

How It Works:

When a model has a thinking pattern configured:

1. Streaming responses are intercepted and transformed
2. Content before end_marker is sent as reasoning_content field
3. Content after end_marker is sent as content field
4. The output follows OpenAI's reasoning_content format for compatibility

Example Output:

// Reasoning content (before end marker)
{"choices": [{"delta": {"reasoning_content": "Let me analyze..."}}]}

// Regular content (after end marker)
{"choices": [{"delta": {"content": "The answer is 42."}}]}

Buffered classification¶

assume_reasoning_first has to guess before it has evidence. It sends the leading tokens out as reasoning_content and waits for </think> to switch channels. A hybrid model that skips its thinking phase and answers directly never emits that marker, so the whole answer arrives on the reasoning channel and clients that render only content show an empty response.

Setting buffered: true replaces the guess with a decision. The router withholds the leading tokens and classifies them from what actually arrives:

models:
  - id: nemotron-3-nano
    metadata:
      thinking:
        pattern: unterminated_start
        end_marker: "</think>"
        assume_reasoning_first: true
        buffered: true
        max_buffer_size: 51200
        reasoning_timeout: "10s"

The decision window starts at the first token, not at request time, and a stream that goes quiet inside the window still resolves on time: the router wakes on the deadline rather than waiting for the next chunk.

buffered is only honored where the blind emission it replaces applies, which is pattern: unterminated_start with assume_reasoning_first: true. On any other pattern it is ignored. It is off by default, so existing deployments keep their current behavior until they opt in.

Time to first token:

Buffering trades TTFT for correct classification. Nothing reaches the client until the decision resolves, so first-token latency becomes the length of the model's thinking phase, bounded by reasoning_timeout. Measured against a mock backend with a 400 ms thinking phase (tests/thinking_buffered_streaming_test.rs):

Streams that never enter the decision window pay nothing: with buffered: false, or on any other pattern, the transformer arms no timer and holds no extra buffer. Keep reasoning_timeout at or below the latency your clients tolerate on an empty screen, and prefer buffered mode for hybrid models where a direct answer is common.

Responses-API-only Models¶

OpenAI exposes some models exclusively via the Responses API (/v1/responses). These models are not reachable through /v1/chat/completions, so a request that targets them on a Chat Completions endpoint returns a 404 not_found from upstream.

The responses_only capability flag marks such models so the router can dispatch them to the Responses API surface instead. The flag defaults to false, so existing model entries do not need to be touched.

Configuration Example:

models:
    - id: gpt-5.4-pro
      metadata:
        display_name: "GPT-5.4 Pro"
        capabilities: ["chat", "vision", "code", "reasoning", "tool"]
        # Served only on /v1/responses; not available on /v1/chat/completions.
        responses_only: true
        limits:
          context_window: 1050000
          max_output: 128000

Models marked Responses-API-only out of the box¶

The list below is kept in sync with model-metadata.yaml and the built-in OpenAI registry (src/infrastructure/backends/openai/models/gpt5_family.rs). When a new Responses-API-only model is added upstream, both files should be updated together.

The flag follows the same lookup priority chain as the rest of the metadata (backend model_configs > model-metadata.yaml > built-in OpenAI metadata), so an operator-supplied entry can override the default for any model.

Marking a new model as Responses-API-only¶

To mark an additional model as Responses-API-only, add responses_only: true to the model entry's metadata block in any of the supported sources. Use the lookup priority that fits the deployment scope:

• model-metadata.yaml for a router-wide default that applies to every backend. Add the flag alongside the existing capability metadata; no other field needs to change. This is the recommended location for newly-released Pro models that are uniformly Responses-API-only across providers.
• Backend model_configs in config.yaml for a backend-specific override (for example, when a self-hosted clone of a Pro model is exposed on a Chat Completions endpoint and should not be dispatched to /v1/responses). A backend-level responses_only: false overrides the metadata-file default for that backend only.
• Built-in OpenAI registry in src/infrastructure/backends/openai/models/gpt5_family.rs for models that ship with the binary. New entries here should also be reflected in model-metadata.yaml so externally-loaded metadata stays consistent.

After updating any of these sources, restart the router or trigger a hot reload so the new flag takes effect on subsequent requests.

Dispatch behavior¶

The router honors responses_only=true on every public surface that would otherwise hit /v1/chat/completions:

• /v1/chat/completions: requests transparently forward to the upstream /v1/responses endpoint and the response is translated back into a strict-mode chat.completion (or chat.completion.chunk for streaming) envelope.
• /anthropic/v1/messages: the Anthropic-formatted request is converted to the Responses API shape, dispatched to /v1/responses, and the upstream response is translated back into Anthropic Messages JSON (or the Anthropic SSE event sequence for streaming). Tool-call round-trips, web-search emulation, and Unix-socket transports all branch on the flag.

In both cases the dispatch is transparent to the client: the request and response shapes match the surface the client called, so no client-side changes are required to use a responses_only model.

For Chat Completions, the bridge preserves compatible GPT-5 controls including reasoning_effort, flat verbosity, metadata, safety_identifier, prompt_cache_key, prompt_cache_options, and prompt_cache_retention. Invalid values fail explicitly instead of being silently omitted during conversion.

Backend-type constraint¶

Only OpenAI and Azure OpenAI backends serve /v1/responses. When a responses_only model is paired with a backend whose type is not OpenAI or Azure OpenAI, the router rejects the request with a 400 invalid_request_error (Anthropic-shaped on /anthropic/v1/messages, OpenAI-shaped on /v1/chat/completions) before any upstream dispatch. The message names both the model and the configured backend type so the misconfiguration is visible from the client log.

The first dispatch per (backend, model) pair logs at info level so operators can confirm Responses-API routing without enabling debug logs. The log line and the responses_bridge_total metric both carry a reason field: responses_only_model for this flag, and tools_with_reasoning for the narrower conditional bridge below.

The narrower flag for function tools with reasoning¶

responses_only is the right flag only when the model has no working Chat Completions route at all. Some models are served on /v1/chat/completions and refuse just one request shape there: function tools combined with a reasoning effort other than none. Setting responses_only on those works, at the cost of sending every request for the model, including plain chat without tools, through the Responses API.

Use chat_completions_tools_require_none_reasoning instead. It bridges only the refused requests and leaves the rest on Chat Completions, and it ships already set on the affected OpenAI models. continuum-router config validate warns when a backend model_configs entry sets responses_only: true on one of them. See Reasoning Effort for the model matrix and the drop-in that clears the flag.

Namespace-Aware Matching¶

The router handles model IDs with namespace prefixes. For example:

• Backend returns: "custom/gpt-5.6-sol", "openai/gpt-5.6-sol", "optimized/gpt-5.6-sol"
• Metadata defined for: "gpt-5.6-sol"
• Result: All variants match and receive the same metadata

This allows different backends to use their own naming conventions while sharing common metadata definitions.

Metadata Priority and Alias Resolution¶

When looking up metadata for a model, the router uses the following priority chain:

1. Exact model ID match
2. Exact alias match
3. Date suffix normalization (automatic, zero-config)
4. Quantization / format suffix normalization (automatic, zero-config; see below)
5. Combined date + format suffix normalization
6. Wildcard pattern alias match
7. Base model name fallback (namespace stripping)

Within each source (backend config, metadata file, built-in), the same priority applies:

1. Backend-specific model_configs (highest priority)

   backends:
     - name: "my-backend"
       model_configs:
         - id: "gpt-5.6-sol"
           aliases: ["gpt-5.6"]
           metadata: {...}  # This takes precedence
   

2. External metadata file (second priority)

   model_metadata_file: "model-metadata.yaml"
   

3. Built-in metadata (for OpenAI and Gemini backends)

Automatic Date Suffix Handling¶

LLM providers frequently release model versions with date suffixes. The router automatically detects and normalizes date suffixes without any configuration:

Supported date patterns:

• -YYYYMMDD (e.g., claude-opus-4-5-20251130)
• -YYYY-MM-DD (e.g., gpt-4o-2024-08-06)
• -YYMM (e.g., o1-mini-2409)
• @YYYYMMDD (e.g., model@20251130)

How it works:

Request: claude-opus-4-5-20251215
         ↓ (date suffix detected)
Lookup:  claude-opus-4-5-20251101  (existing metadata entry)
         ↓ (base names match)
Result:  Uses claude-opus-4-5-20251101 metadata

This means you only need to configure metadata once per model family, and new dated versions automatically inherit the metadata.

Automatic Quantization and Format Suffix Handling¶

Real-world model IDs arriving at /v1/models, routing logic, and backend metadata enrichment frequently combine a canonical base ID with one or more trailing quantization, format, or flavor tokens. The router strips an allowlisted set of such tokens iteratively and retries exact-id, exact-alias, and date-suffix matching after each peel, so you only need to configure metadata for the canonical base ID.

Token Categories¶

The following trailing tokens are detected and stripped (case-insensitive):

Parameter-Count Suffixes are Preserved¶

Tokens that look like parameter counts are never stripped, even when they share a trailing b:

• Kept: -32b, -70b, -8b, -4b, -a3b, -a22b, -0.6b, -1.7b, -e4b
• Stripped: -4bit, -8bit, -16bit (the literal bit suffix marks quantization)

This discrimination ensures that a parameter-count variant like qwen3-32b resolves only to explicit qwen3-32b metadata, never to a generic qwen3 entry via accidental stripping.

Layered Peeling¶

Tokens are stripped one at a time. After each peel, the router re-runs exact-id, exact-alias, and date-suffix match before attempting another peel. This lets alias configurations like gemma-3-4b-it-qat still win even when the request is gemma-3-4b-it-qat-4bit:

Request: gemma-3-4b-it-qat-4bit
         ↓ (peel -4bit)
Try:     gemma-3-4b-it-qat
         ↓ (matches alias of gemma-3-4b-qat)
Result:  Uses gemma-3-4b-qat metadata

A peel that ends on a date suffix is also date-stripped and fed back through the peel loop, so a trailing date does not terminate the chain:

Request: Ministral-3-14B-Instruct-2512
         ↓ (strip the -2512 date)
Try:     ministral-3-14b-instruct
         ↓ (peel -instruct)
Result:  Uses ministral-3-14b metadata

Priority Note¶

Stripping runs after exact-id and exact-alias match. A canonical base ID that happens to end in an allowlisted token (for example gemma-3-12b-qat) wins before the peel phase runs, so existing configurations remain stable.

Suffix-Order Ambiguity¶

Both -qat-4bit and -4bit-qat orderings appear in real-world model IDs. Peeling removes one token at a time from the right, so the intermediate form mirrors the order in which tokens appeared in the input. The match sequence for gemma-3-12b-qat-4bit is gemma-3-12b-qat-4bit → gemma-3-12b-qat → gemma-3-12b, while gemma-3-12b-4bit-qat goes gemma-3-12b-4bit-qat → gemma-3-12b-4bit → gemma-3-12b. If both suffix orderings need to resolve to the same QAT-variant metadata, configure the canonical QAT base ID (gemma-3-12b-qat) with the matching metadata and let the non-QAT form (gemma-3-12b) carry its own entry; the deepest successful match wins at each peel depth. When the QAT and non-QAT variants need distinct tier or capability metadata, prefer aliases that enumerate the reorderings over relying on the peel order alone.

Length Bounds¶

The layered peel phase caps input length at 256 characters and iteration count at 8 peels as defense-in-depth against pathological inputs. Matching still runs (the exact-id and exact-alias phases remain in effect), but the peel phase short-circuits instead of walking a long allowlist-token chain. Request handlers enforce the same 256-character cap on the model field for every chat / completion / embedding endpoint, so normal traffic never hits the internal cap.

Case Insensitivity¶

Stripping is case-insensitive, so Qwen3.5-4B-4bit, QWEN3.5-4B-4BIT, and qwen3.5-4b-4bit all resolve to the same qwen3.5-4b metadata entry. Exact-id and exact-alias match phases (1 and 2) remain case-sensitive, so HuggingFace-style aliases like BAAI/bge-m3 keep their original behavior.

Wildcard Pattern Matching¶

Aliases support glob-style wildcard patterns using the * character:

• Prefix matching: claude-* matches claude-opus, claude-sonnet, etc.
• Suffix matching: *-preview matches gemini-3.1-pro-preview, o1-preview, etc.
• Infix matching: gpt-*-turbo matches gpt-4-turbo, gpt-3.5-turbo, etc.

Example configuration with wildcard patterns:

models:
  - id: "claude-opus-4-5-20251101"
    aliases:
      - "claude-opus-4-5"     # Exact match for base name
      - "claude-opus-*"       # Wildcard for any claude-opus variant
    metadata:
      display_name: "Claude Opus 4.5"
      # Automatically matches: claude-opus-4-5-20251130, claude-opus-test, etc.

  - id: "gpt-5.6-sol"
    aliases:
      - "gpt-5.6"             # Exact match for the catalog alias
      - "gpt-5.6-*"           # Wildcard for any GPT-5.6 variant
    metadata:
      display_name: "GPT-5.6 Sol"

Priority note: Exact aliases are always matched before wildcard patterns. When both could match, the exact alias wins.

Alias Dispatch¶

Metadata resolution decides which entry supplies pricing, capabilities, and the /v1/models listing. Backend selection is separate and matches the requested name literally against each backend's models: list and the aggregated live catalog. The two agree as long as every configured name is one the backend serves. When they disagree, the router rewrites the request before selection, on every ingress that picks a backend from a client-supplied model name. All four conditions must hold:

1. The requested name is an exact alias (or a models/-prefixed spelling of one) of a different canonical id, declared in an enabled backend's model_configs or in model-metadata.yaml. The fuzzy phases of the priority chain above (date suffix, quantization suffix, HuggingFace prefix, wildcard aliases) are not consulted: they decide which entry supplies metadata, and dispatch identity is a stronger claim.
2. Every enabled backend whose models: list names the requested id has at least one entry in the live catalog. A backend whose discovery has not completed contributes nothing, and its silence is not evidence.
3. No backend serves the requested name in the live catalog (the ids the backend enumerated on GET /v1/models, or the configured names for a backend without discovery).
4. A user-routable backend serves the canonical id in the live catalog.

Only then is the model field replaced with the canonical id. A name that any backend serves literally is never touched, so a local engine that lists unsloth/Qwen3.6-35B-A3B-GGUF keeps receiving that exact id. A request nothing can serve keeps its original name and fails as it did before. The upstream answers with the id it served, so a client that asked for the alias sees the canonical id in the response model field, which is how OpenAI itself answers a floating name.

Three interactions are worth knowing. The per-key allowed_models list and the request_params model scopes are judged on the name the client asked for; a key that admits the alias keeps working after the rewrite. fallback.fallback_chains is looked up under the dispatched name, so a chain keyed on an alias will not fire; key it on the canonical id, and the router logs a warning at request time when it finds a chain keyed on an alias it is rewriting. Hub-attributed requests, requests under a Hub exact-model budget guard, and AppProxy ingress-pinned requests are never rewritten, since each already carries a model decision.

The motivating case is gemini-3.1-pro: model-metadata.yaml declares it as an alias of gemini-3.1-pro-preview because Google serves only the -preview id, so a request for gemini-3.1-pro is dispatched as gemini-3.1-pro-preview instead of reaching Google as a 404. The rewrite is logged at info level as Dispatching a model alias as the canonical id its backend serves, with the requested and canonical names.

The covered ingresses are /v1/chat/completions, /anthropic/v1/messages, /anthropic/v1/messages/count_tokens, /v1/responses, /v1/responses/compact, /v1/embeddings including its native Gemini multimodal sub-path, /v1/images/generations, /v1/images/edits, /v1/images/variations, and the realtime handshake. ACP is the only exception and stays literal-only, because ACP usage and the ACP architecture promise literal matching.

Two details are worth knowing about the covered set. A 404 or 403 raised after a rewrite names the model the client asked for, not the id the router resolved to. And the image edit and variations endpoints accept a fixed list of client-facing model names, checked before the rewrite, so an alias reaches them only when it is itself one of those names.

Using Aliases for Model Variants¶

Aliases are particularly useful for:

• Different quantizations: qwen3-32b-i1, qwen3-23b-i4 → all use qwen3 metadata
• Version variations: gpt-4-0125-preview, gpt-4-turbo → share gpt-4 metadata
• Deployment variations: llama-3-70b-instruct, llama-3-70b-chat → same base model
• Dated versions: claude-3-5-sonnet-20241022, claude-3-5-sonnet-20241201 → share metadata (automatic with date suffix handling)

Example configuration with aliases:

model_configs:
  - id: "qwen3"
    aliases:
      - "qwen3-32b-i1"     # 32B with 1-bit quantization
      - "qwen3-23b-i4"     # 23B with 4-bit quantization
      - "qwen3-16b-q8"     # 16B with 8-bit quantization
      - "qwen3-*"          # Wildcard for any other qwen3 variant
    metadata:
      display_name: "Qwen 3"
      summary: "Alibaba's Qwen model family"
      # ... rest of metadata

Aliases vs. suffix normalization: when to use which¶

Two coverage layers resolve non-canonical model ids to their owning metadata entry: explicit YAML aliases, and the layered suffix-peel allowlist in src/models/pattern_matching.rs. They are complementary, not redundant. This section explains how to choose between them when adding a new entry.

Matching phase order¶

The pipeline runs in this order, and a successful match in an earlier phase short-circuits the later ones:

1. Exact model id (case-sensitive).
2. Exact alias (case-sensitive).
3. Date-suffix normalization (-YYYYMMDD, -YYYY-MM-DD, -YYMM, @YYYYMMDD).
4. Layered quantization / format / flavor peel (case-insensitive; after each peel, exact-id + exact-alias + date-suffix phases re-run; combined date + format handled in the same loop).
5. HuggingFace repo-prefix stripping (vendor/repo -> repo) with re-entry into phases 1-4 on the stripped residual. Single-hop re-entry; phase 5 does not recurse.
6. Wildcard alias (glob-style * patterns).

A retained explicit alias runs in phase 2, strictly before the peel (phase 4) and before the prefix-strip layer (phase 5). When a retained alias and a peel-or-strip path would resolve to different metadata, the alias wins deterministically. Aliases are therefore a stronger intent signal than peel-or-strip coverage, not a weaker one.

The three alias classes¶

Every alias in model-metadata.yaml falls into one of three classes.

peel-coverable¶

Normalization reaches the same owner id without the alias, and the target metadata is the correct one. These are deletion candidates. Example: qwen3.6-35b-a3b-instruct as an alias of qwen3.6-35b-a3b. Phase 4 peels the FLAVOR token -instruct and lands on the base id directly, so the explicit alias adds no coverage. When deleting an alias in this class, add a regression assert in tests/format_suffix_normalization_test.rs::real_metadata_removed_aliases_still_resolve so the peel coverage that replaces it stays locked in.

vendor-prefix¶

The alias carries a vendor or repo prefix that suffix peel cannot strip, because peel only removes right-side tokens from a closed allowlist. Phase 5 (HuggingFace prefix stripping with re-entry into phases 1-4, where phase 4 is case-insensitive) resolves the mixed-case HF form even without the alias. Example: Qwen/Qwen3.6-35B-A3B as an alias of qwen3.6-35b-a3b. Phase 2 wins on the explicit alias, but phase 5 would also reach the base id via Qwen/ -> residual Qwen3.6-35B-A3B -> phase 4 case-insensitive match. These aliases are peel-coverable-adjacent: phase 5 makes them redundant for resolution, but they remain a deterministic intent signal. Keep them with a YAML comment noting the covering phase.

intentional-override¶

The alias deliberately routes a differently-weighted model under another entry's metadata, as an operator decision. Keep. Example: smoothie-qwen3-32b-i1 as an alias of smoothie-qwen3. The smoothie-qwen3-32b-i1 fine-tune has its own weights; the operator has chosen to surface it under the umbrella smoothie-qwen3 metadata rather than give it a dedicated entry. Peel must not infer this equivalence on its own. When an alias sits in this class, the YAML comment on the line must note that the underlying weights differ from the owner id, distinguishing an intentional override from a mechanical normalization gap.

Guidance for adding a new alias¶

Before adding a line to model-metadata.yaml, ask whether peel already covers it.

• If the new id is a canonical base with a trailing quantization, format, or flavor token already in the allowlist, and the weights are equivalent to the base metadata, do not add the alias. The peel handles it, and adding the alias would be dead code.
• If the new id shares weights with the base but ends in a token class that peel does not handle (for example, a novel fine-tune label like -abliterated or a new quantization format like -nf4), prefer extending the peel allowlist in src/models/pattern_matching.rs. This is a code change with test coverage, and it covers the entire token class in one move.
• If the new id has a vendor prefix, a repo namespace that normalization would not case-match, a parameter-count token blocking the peel chain (-Nb, -aNb, -eNb), or intentionally-different weights, add the alias with a YAML comment that states the reason. If weights differ from the owner id, say so in the comment.

Surface distinction: code-gated vs. YAML-gated¶

The peel allowlist is the strategic layer. Aliases are the tactical override and emergency channel.

Token categories already on the peel allowlist¶

The allowlist in src/models/pattern_matching.rs currently covers:

• BIT_WIDTH: -2bit, -3bit, -4bit, -5bit, -6bit, -8bit, -16bit
• GGUF_QUANT: -Q4_K_M, -Q4_K_S, -Q5_K_M, -Q6_K, -Q8_0, -Q2_K, -IQ2_XS, -IQ3_XXS, -IQ4_XS, -F16, -F32, -BF16
• FP_FORMAT: -FP4, -FP8, -FP16, -FP32, -NVFP4, -MXFP4, -MXFP8
• INT_FORMAT: -INT2, -INT4, -INT8
• WEIGHT_ACTIVATION: -W<bits>A<bits> (-W4A16, -W8A8, -W4A8), also accepted with the quantized. prefix (-quantized.w4a16)
• SCALING_GRANULARITY: -block, -dynamic, -static, peeled only after an FP / INT / weight-activation token
• LIBRARY: -AWQ, -GPTQ, -BNB, -HQQ, -EXL2, -EXL3, -MLX
• IMATRIX: -i1 through -i8, -q2 through -q8
• UNSLOTH: -UD-Q<digit>_<KIND>, -UD-IQ<digit>_<KIND>, -unsloth
• CONTAINER: -GGUF, -GGML, -SAFETENSORS, -ONNX
• FLAVOR: -it, -instruct, -chat, -base, -thinking, -qat
• HARDWARE: -rngd, -warboy, -atom, -atommax, -rebel

Tokens that mark a behavior-changing derivative are deliberately absent, because the peeled base would be a different logical model: -abliterated, -heretic, -uncensored, -distilled, -REAP-<params>, -DFlash / -DSpark, -speculator.eagle3, -MTP, -assistant.

Parameter-count suffixes (-Nb, -aNb, -eNb, -0.6b, -1.7b) are never peeled. They are part of canonical model identity and terminate the peel chain. This is why qwen3-32b-i1 must be kept as an explicit alias of qwen3: phase 4 strips -i1 and then halts at -32b, so without the alias the chain exhausts before reaching the base id.

HuggingFace repo-prefix stripping (phase 5)¶

Phase 5 normalizes HuggingFace-style vendor/repo prefixes off the left side of a model id, complementing the right-side suffix peel. It resolves the common HF-GGUF class where a user submits an id like unsloth/Qwen3.6-35B-A3B-GGUF and expects it to route to the canonical qwen3.6-35b-a3b metadata, without an explicit alias for every vendor x base x quant combination.

How phase 5 runs¶

1. The input is inspected for a / separator. No /, no-op.
2. Total segments (count of / plus one) must be at most MAX_PREFIX_SEGMENTS (3). org/team/repo is permitted; a/b/c/d/model is rejected outright.
3. All segments must be non-empty and free of ASCII whitespace. Malformed inputs like /repo, vendor/, vendor//repo, or vendor /repo are rejected.
4. On success, the residual is the substring after the last /. This residual is fed back into phases 1-4 with the re-entry gate closed. Phase 5 does not recurse: the inner call cannot trigger phase 5 again, so the recursion depth is exactly 1 by construction.

Composition with suffix peel¶

The re-entry runs through phase 4, so prefix stripping composes with suffix peel in a single lookup. unsloth/Qwen3.6-35B-A3B-GGUF strips to Qwen3.6-35B-A3B-GGUF, phase 4 peels -GGUF, case-insensitively matches qwen3.6-35b-a3b. This is the motivating case for the phase and covers HuggingFace GGUF forks without requiring hand-enumerated aliases.

Registered-alias precedence¶

Operators who explicitly register a vendor/repo form as a YAML alias keep deterministic control. Phase 2 runs before phase 5, so the exact alias wins before the stripping layer ever considers the input. Use this when the prefixed form must route to a different metadata entry than the canonical base id would.

Out of scope¶

• Hyphen-delimited vendor prefixes (e.g., smoothie-qwen/smoothie-qwen3-32b-i1). Different semantic class, different detection difficulty, and often represents different weights where silent base-metadata routing is the wrong call.
• Automatic vendor discovery from the HuggingFace API. The layer is purely syntactic.
• Extending the suffix peel allowlist. Orthogonal change; follow the peel-extension path for novel token classes.

Security bounds¶

Parallels the suffix peel:

Phase 5 is constant-time on adversarial input: after the segment-count, emptiness, whitespace, and length guards, the work reduces to a single slice lookup plus one additional pass through phases 1-4.

Audit procedure¶

To re-audit the YAML, run:

cargo test --test alias_audit_helper -- --ignored --nocapture audit_metadata_aliases

The helper prints every alias with its classification (REDUNDANT, LOAD-BEARING-DRIFT, LOAD-BEARING-LOSS, or WILDCARD) and the post-removal resolution target.

API Response¶

The /v1/models endpoint returns enriched model information:

{
  "object": "list",
  "data": [
    {
      "id": "gpt-5.6-sol",
      "object": "model",
      "created": 1234567890,
      "owned_by": "openai",
      "backends": ["openai-proxy"],
      "metadata": {
        "display_name": "GPT-5.6 Sol",
        "summary": "Frontier GPT-5.6 model for complex professional work",
        "capabilities": ["chat", "vision", "code", "reasoning", "tool"],
        "knowledge_cutoff": "2026-02",
        "pricing": {
          "input_tokens": 5.0,
          "output_tokens": 30.0
        },
        "limits": {
          "context_window": 1050000,
          "max_output": 128000
        }
      }
    }
  ]
}

Request Parameter Policy¶

Use the optional request_params section to define typed defaults, operator-selected overrides, and limits for the request-parameter policy. The Router validates and hot-reloads this configuration snapshot, then applies it once at Chat Completions, Completions, Responses, and Anthropic Messages ingress before cache identity or provider routing is chosen.

request_params:
  defaults:
    max_tokens: 1024
  overrides:
    temperature: 0.0
  limits:
    max_tokens:
      min: 1
      max: 4096
  models:
    gpt-5:
      defaults:
        top_p: 0.9
      limits:
        max_tokens:
          max: 2048

Maintained examples use block-style YAML. Equivalent flow mappings such as { max_tokens: 1024 } are also accepted, and TOML configuration has the same semantics.

Evaluation Order¶

Protocol integrations apply the three modes in a fixed order:

1. defaults fills a missing or JSON/YAML-null field and preserves a value supplied by the client.
2. overrides replaces the resulting value, whether it came from the client or a default.
3. limits clamps an existing value to the inclusive min and max bounds. A limit does not create a missing field.

The top-level objects are the global policy. Each models.<name> entry may define the same three objects. Model fields override global fields independently within each mode; fields omitted from a model entry inherit the corresponding global value or range.

The runtime contract requires model names to match exactly after local alias resolution. It also requires the selected scope to be fixed before control-plane substitution, arbitrage, backend selection, retry, or fallback and remain unchanged across retries and fallbacks. Hub tier limits delivered through control_plane.policy are limits-only and intersect these local limits per bound, so neither side can broaden the other; an empty intersection rejects the request. Backend scopes, regular expressions, and glob matching are unsupported.

Supported Parameters and Validation¶

Only the following canonical parameter names are accepted:

Unknown modes or parameter names, wrong scalar types, non-finite floats, out-of-range values, and min > max are rejected with a path-qualified error. models accepts at most 64 exact scopes; each model key must be non-empty after trimming, at most 256 bytes, and free of control characters. The same parameter may appear in more than one mode because the evaluation order is deterministic. A null policy value is treated as omitted; it is not a request to force JSON null.

Runtime Provider Compatibility¶

Compatibility checks apply only to fields that the policy configured and actually changed or constrained. Client fields that the policy did not govern retain their existing provider-specific passthrough behavior. If an eligible primary or configured fallback route cannot preserve a policy-effective field, the Router rejects the request before cache lookup or upstream I/O with unsupported_request_parameter instead of silently dropping the enforced value.

Streaming fallback follows the same matrix. A pre-stream hop re-enters the typed per-backend dispatch, so a hop onto a native Anthropic, Bedrock, or Gemini backend is served through that provider's native pipeline with the single wire-format conversion at dispatch. Once SSE output has begun, mid-stream recovery speaks only the OpenAI-compatible wire: a chain entry that resolves to a native-protocol or Unix-socket backend is skipped deterministically in favor of the next entry, and is never forwarded as a partly translated policy request. Each hop is rebuilt from the canonical effective request, so a multi-hop fallback cannot accumulate conversions or lose a restored field.

Response-cache lookup and storage are enabled only when routing resolves to one policy-compatible backend and no fallback chain can change that destination. The backend identity is included in the cache namespace. Requests with multiple eligible backends or an enabled fallback chain bypass exact, prefix, and streaming caches so an entry produced under one provider cannot satisfy another route.

Observability and Enforcement¶

When policy changes at least one request field and the selected provider path can preserve the effective request, the Router adds the informational x-continuum-request-params response header. Its comma-separated tokens contain only a bounded action and canonical parameter name, such as default:max_tokens or limit:temperature; request values, model names, keys, and tier identifiers are never included.

The policy is mandatory operator configuration. Clients cannot disable or bypass it with a request header, query parameter, or request-body option. The informational response header does not provide an opt-out and is not an authorization credential.

The header is present on successful responses, compatible cache hits, and upstream error responses produced after an effective mutation. It is omitted when policy is a no-op and when policy or provider compatibility rejects the transaction. A rejected transaction is rolled back and never receives a misleading applied header. Logs and Prometheus metrics follow the same bounded applied/rejected outcomes.

Hot Reload and Privacy¶

request_params is an immediate hot-reload section. In control-plane builds with control_plane.config_sync.enabled: true, Continuum Hub may also manage the public kv_routing leaves for selection_strategy, prefix_routing.*, and the bounded kv_cache_index.* routing schema; set request_params_immutable: true or kv_routing_immutable: true to pin either whole section locally. A successfully parsed and validated revision atomically replaces the published configuration snapshot. A rejected revision never replaces the last-known-good snapshot. Each request captures one immutable config snapshot and one authenticated Hub tier policy before mutation, so in-flight requests, retries, and fallbacks do not change policy halfway through.

Applied-policy telemetry exposes only bounded parameter names, actions, composition source, and outcome, never request values. Config and MCP views may show the operator-authored policy, but they do not serialize prompt content or arbitrary request data and continue to redact secret-typed config fields.

Hot Reload¶

When the hot-reload feature and file watcher are active, a valid file revision is published as a new immutable configuration snapshot. Invalid revisions leave the previous snapshot active. Not every parsed field has a live runtime consumer, so use the conservative matrix below.

Applied without restart¶

• Backend additions/removals/edits (the pool is reconciled; new backends receive an immediate health check)
• Health-check intervals and thresholds
• Rate-limit and circuit-breaker policies when their services are active
• Global-prompt, request-parameter, and streaming policy read by new requests
• Fallback chain/policy edits when the fallback service already exists
• Smart-routing policy snapshots
• selection_strategy (swapped onto the live pool atomically; membership, stats, and in-flight accounting are preserved)
• Retry policy (new requests capture the updated complete policy; in-flight loops retain their entry snapshot)
• Per-request, streaming, image, model-override, and health-check timeout budgets
• All prefix_routing.* fields: enabled and max_prefix_length for request-time prefix extraction; load_factor_epsilon and virtual_nodes swapped onto the live pool (a virtual_nodes change also rebuilds the hash ring); anthropic_cache_control_injection and salt_echo read per-request
• routing.engine_load.max_staleness_intervals and routing.engine_load.admission.* (applied by the engine-stats configuration watcher on each published reload)

Restart required or currently not live¶

• server.bind_address, server.socket_mode, and server.workers
• server.connection_pool_size and CORS router layers
• Logging subscriber format/filter and tracing middleware construction
• Toggling optional services that were absent at startup, including fallback.enabled
• Response-cache backend/object construction
• timeouts.connection and timeouts.request.streaming.total (the shared HTTP-client connection/overall ceilings; request budgets still reload live)
• routing.engine_load scoring fields (enabled, engine_load_weight, balance_abs_threshold, balance_rel_threshold): the KV overlap scorer captures them at construction

After editing any restart-required field, validate and restart:

continuum-router config validate config.yaml

Check watcher availability and the router's capability summary through the authenticated Admin API:

curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8080/admin/config/hot-reload-status

Treat the active subsystem state, rather than the newly parsed file alone, as authoritative when a restart-required value changes.

Distributed Tracing¶

Continuum Router supports distributed tracing for request correlation across backend services. This feature helps with debugging and monitoring requests as they flow through multiple services.

Everything in this section is about header propagation: which identifiers the router reads from a request and which it forwards to backends. It is always active and needs no special build. To additionally export the router's own spans to an OpenTelemetry collector, configure the tracing.otlp subsection documented in Distributed Trace Export. That subsection is off by default and requires a binary built with the otel Cargo feature; leaving it out changes nothing about the behavior described here.

Configuration¶

tracing:
  enabled: true                         # Enable/disable distributed tracing (default: true)
  w3c_trace_context: true               # Support W3C Trace Context header (default: true)
  headers:
    trace_id: "X-Trace-ID"              # Header name for trace ID (default)
    request_id: "X-Request-ID"          # Header name for request ID (default)
    correlation_id: "X-Correlation-ID"  # Header name for correlation ID (default)

How It Works¶

1. Trace ID Extraction: When a request arrives, the router extracts trace IDs from headers in the following priority order:
2. W3C traceparent header (if W3C support enabled)
3. Configured trace_id header (X-Trace-ID)
4. Configured request_id header (X-Request-ID)

5. Configured correlation_id header (X-Correlation-ID)

6. Trace ID Generation: If no trace ID is found in headers, a new UUID is generated.

7. Header Propagation: The trace ID is propagated to backend services via multiple headers:

8. X-Request-ID: For broad compatibility
9. X-Trace-ID: Primary trace identifier
10. X-Correlation-ID: For correlation tracking
11. traceparent: W3C Trace Context (if enabled)

12. tracestate: W3C Trace State (if present in original request)

13. Retry Preservation: The same trace ID is preserved across all retry attempts, making it easy to correlate multiple backend requests for a single client request.

Structured Logging¶

When tracing is enabled, all log messages include the trace_id field:

{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "info",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "message": "Processing chat completions request",
  "backend": "openai",
  "model": "gpt-5.6-sol"
}

W3C Trace Context¶

When w3c_trace_context is enabled, the router supports the W3C Trace Context standard:

• Incoming: Parses traceparent header (format: 00-{trace_id}-{span_id}-{flags})
• Outgoing: Generates new traceparent header with preserved trace ID and new span ID
• State: Forwards tracestate header if present in original request

Example traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

Disabling Tracing¶

To disable distributed tracing:

tracing:
  enabled: false

Load Balancing Strategies¶

Use the top-level selection_strategy field. The six implemented values are RoundRobin, WeightedRoundRobin, LeastLatency, Random, ConsistentHash, and PrefixAwareHash. Backend health filtering happens before selection.

selection_strategy: WeightedRoundRobin

backends:
  - name: large
    url: http://large.example.com
    weight: 3
  - name: small
    url: http://small.example.com
    weight: 1

See Load Balancing for exact semantics and current startup/hot-reload limitations.

Engine-Load-Aware Selection¶

The routing.engine_load section (issue #1447) turns the engine_stats snapshots into a selection signal. It extends the KV overlap scorer, so it needs prefix_routing.enabled: true plus an enabled kv_cache_index for the scorer to exist, and engine_stats.enabled: true for data; continuum-router config validate warns when either dependency is missing.

routing:
  engine_load:
    enabled: false                # Scoring term switch (restart to apply)
    engine_load_weight: 0.3       # Additive scorer weight, 0.0-1.0 (restart)
    max_staleness_intervals: 3    # Snapshot freshness bound in polling
                                  # intervals, 1-100 (hot-reload, gradual)
    balance_abs_threshold: 64     # Waiting-request spread (max-min) that must
                                  # be exceeded before the term ranks (restart)
    balance_rel_threshold: 1.5    # Waiting-request ratio (max/min, >= 1.0)
                                  # that must ALSO be exceeded (restart)
    admission:
      enabled: false              # Saturation admission hint (hot-reload)
      kv_usage_threshold: 0.98    # KV usage refusal threshold, 0.0-1.0

The scoring term prefers, among the prefix holders, the backend whose engine reports fewer waiting_requests (normalized by total_slots when every fresh candidate states one, otherwise by the request candidate set's maximum waiting_requests) and lower kv_cache_usage. A snapshot older than engine_stats.interval * max_staleness_intervals is treated as absent: that candidate keeps its base score, and when every candidate is stale the pass matches the base scoring exactly. The two balance thresholds form a hysteresis dead band mirroring the SGLang Model Gateway's balance gates (both must be exceeded before the term ranks), so oscillating queues cannot flap selection. The gate, normalization, and scorer cache are isolated by the exact live candidate set. Decisions are counted in routing_engine_load_decisions_total{backend,reason}; see Metrics.

The admission hint runs at the shared selection seam independently of the scorer: when every healthy candidate has a fresh snapshot with kv_cache_usage above the threshold, selection is refused instead of queueing. On the chat path the refusal is a retryable 503 that participates in fallback.fallback_chains through the BackendUnhealthy trigger, so a configured chain takes over instead of the client seeing the error; a candidate with no fresh statement always admits.

Per-Backend Retry Configuration¶

backends:
  - name: "slow-backend"
    url: "http://slow.example.com"
    retry_override:               # Complete backend-specific retry policy
      max_attempts: 5
      initial_delay: "500ms"
      max_delay: "60s"
      backoff_multiplier: 2.0
      jitter: true
      retryable_status_codes: [429, 502, 503, 504]
      retryable_errors: [ConnectionError, TimeoutError]
      timeout: "120s"

Model Fallback¶

Configure ordered alternative models with the top-level fallback section:

fallback:
  enabled: true
  mid_stream_enabled: true
  fallback_chains:
    gpt-5.4:
      - gpt-5.4-mini
      - claude-sonnet-4-6
  fallback_policy:
    trigger_conditions:
      error_codes: [429, 500, 502, 503, 504]
      timeout: true
      connection_error: true
      model_not_found: true
      backend_unhealthy: true
    max_fallback_attempts: 3
    fallback_timeout_multiplier: 1.5
    preserve_parameters: true
    max_concurrent_dials_per_backend: 50
  model_settings:
    gpt-5.4:
      fallback_enabled: true
      notify_on_fallback: true

The regular proxy path does not produce circuit_breaker_open; use active health checks and backend_unhealthy for inference traffic. See Model Fallback for validation, headers, metrics, cross-provider translation, and the distinction between pre-stream and mid-stream behavior.

Mid-Stream Fallback¶

Mid-stream fallback allows the router to transparently continue an active SSE stream on a fallback backend when the primary backend fails mid-response. The client's connection remains open and sees an uninterrupted response with only a brief pause during the switchover.

Mid-stream fallback activates automatically when fallback.enabled: true and a fallback chain is configured for the requested model. The streaming.mid_stream_fallback section controls how the fallback backend is invoked (continuation vs restart mode), not whether fallback happens.

Disabling Mid-Stream Buffering¶

fallback.mid_stream_enabled (default true) decouples pre-stream fallback from mid-stream buffering. When set to false while fallback.enabled: true and a chain is configured:

• The initial connection still re-routes through the fallback chain on connection errors, timeouts, or trigger error codes (pre-stream fallback is unaffected).
• Once the SSE stream starts, no stream accumulator is allocated, so there is no per-stream buffering or its associated memory cost.
• A failure that happens after the stream has started surfaces to the client as a normal stream error instead of triggering a transparent switch.

Use mid_stream_enabled: false on memory-constrained hosts, or under high concurrency with many long-context streaming sessions where the ~100-200 KB per-stream buffer adds up.

This flag is distinct from streaming.mid_stream_fallback.enabled:

fallback:
  enabled: true             # Pre-stream fallback stays on
  mid_stream_enabled: false # No per-stream buffering; mid-stream errors reach the client
  fallback_chains:
    "gpt-5.6-sol":
      - "gpt-5.6-terra"
      - "gpt-5.6-luna"

Configuration¶

fallback:
  enabled: true  # Required: enables mid-stream fallback path
  fallback_chains:
    "gpt-5.6-sol":
      - "gpt-5.6-terra"
      - "gpt-5.6-luna"

streaming:
  mid_stream_fallback:
    # Enable continuation mode (default: true).
    # When true, accumulated partial response is used to build a continuation prompt,
    # producing uninterrupted output for the client.
    # When false, the fallback backend restarts the request from scratch, which may
    # cause duplicate or incoherent content if partial output was already sent.
    enabled: true

    # Minimum estimated tokens accumulated before using continuation mode (default: 50)
    # Below this threshold the request is restarted from scratch on the fallback backend
    # instead of appending a continuation prompt.
    min_accumulated_tokens: 50

    # Maximum hops after SSE output has begun (default: 2, max: 10). A hop
    # before the first chunk follows fallback.fallback_policy.max_fallback_attempts.
    max_fallback_attempts: 2

    # Prompt appended as a user message after the partial assistant response
    continuation_prompt: "Continue from where you left off exactly. Do not repeat any previously generated content."

How It Works¶

1. The client sends a streaming chat completion request.
2. The router begins streaming from the primary backend, accumulating response content.

3. If the backend fails mid-stream (connection drop, timeout, error event):

   • The error is NOT forwarded to the client.
   • The accumulated partial response is captured.
   • The next healthy backend in the fallback chain is selected (unhealthy backends are skipped).
   • A continuation or restart request is sent to the fallback backend.
   • Streaming resumes on the fallback backend without closing the client connection.

4. The client receives an uninterrupted response with only a brief pause during the switchover.

Continuation vs. Restart Mode¶

The min_accumulated_tokens threshold controls which recovery mode is used:

Continuation mode (the default) produces uninterrupted output for the client. Restart mode is used automatically when there is too little context to continue meaningfully, or when the accumulated response is too long to include safely. Explicitly setting enabled: false forces restart mode unconditionally, which may cause duplicate or incoherent content visible to the client.

Edge Case Handling¶

The mid-stream fallback path addresses several edge cases automatically:

• Global timeout budget: All fallback attempts share the original request start time. Each attempt checks remaining budget before sending, preventing indefinite timeout accumulation across the chain.
• Canonical cross-provider payload: When the fallback model is on a different provider (e.g., OpenAI → Anthropic), the request stays in canonical OpenAI chat-completions form. The hop swaps the model name and nothing else; wire-format conversion, and every decision about which fields may reach the wire, happens once at dispatch, keyed on the selected backend's configured type.
• Concurrent request storms: When a primary goes down, every in-flight request headed for it fails at once and every one of them hops onto the same rescue backend. fallback.fallback_policy.max_concurrent_dials_per_backend (default 50, 0 = unlimited, range 0..=10000) bounds that herd per backend. The bound covers a dial only: a permit is taken just before a hop's outbound request is sent and released as soon as the provider handshake returns (status and headers, or the transport error), never across the response body, because the body phase is already bounded by server.max_concurrent_requests, the timeouts, and the circuit breaker, and a whole-stream hold would turn an outage into an availability cliff. It applies on all four fallback paths: the non-streaming funnel (every retry attempt of a hop dials again and is bounded again), the pre-stream connection phase in both mid_stream_enabled modes, a hop onto a native-protocol backend, and the hops the relay performs after the SSE response is committed. Only hops are bounded: the primary attempt never takes a permit, and neither does a request whose backend selection walked onto a fallback model before the first dial, because in that state the primary is already known to be down and the rescue backend is effectively the primary, so capping it would cap steady-state throughput. A saturated backend queues the dial FIFO for at most the effective timeouts.connection (and, on the streaming paths, at most what is left of the cross-attempt chain budget); on expiry the hop fails for that backend with a backend_unhealthy trigger (on by default), so the chain advances to the next entry or exhausts with the preserved upstream failure. The permit is taken before the circuit-breaker admission, so a saturation timeout never consumes a half-open probe slot and records nothing on the breaker. Two exceptions hold the permit for the whole call rather than the dial, because their transports expose no handshake seam: a non-streaming hop over a Unix socket and a non-streaming hop onto a Bedrock runtime backend. A saturation timeout on a hop onto a native-protocol backend advances the chain on backend_unhealthy like any other hop, because a native attempt that fails before the provider answers joins the same chain walk (see Model fallback). The setting reloads immediately: a changed limit resets every per-backend semaphore, so new dials use the new size while dials already holding or queued on an old one keep it and release into it harmlessly, which means the bound for one backend is momentarily the sum of the old and new in-flight permits while a change is in flight. Saturation timeouts are counted in fallback_dial_bound_saturated_total{backend}.
• Accumulator truncation: When accumulated response content exceeds 100 KB, the continuation mode is forced to restart to avoid sending incoherent context to the fallback backend.
• Health re-check: Backend health is re-verified before each fallback attempt in the chain. Unhealthy backends are skipped to the next entry.
• Missing [DONE] marker: Streams ending without [DONE] but with finish_reason: "stop" are treated as completed successfully, preventing unnecessary fallback.

Metrics¶

Three Prometheus metrics track mid-stream fallback activity. See Mid-Stream Fallback Metrics for details.

Minimizing Failover Latency¶

When a backend goes down during streaming, the time until the fallback backend takes over depends on several configuration parameters across different subsystems. Below is a tuning guide for minimizing this switchover delay.

How failover delay is composed¶

The total time a client waits during a mid-stream failover is roughly:

failover_delay ≈ failure_detection_time + health_recheck_time + fallback_connection_time

Each component maps to specific configuration:

Recommended configuration for fast failover¶

# 1. Timeouts — the most impactful settings for failover speed
timeouts:
  connection: 5s               # Faster TCP connect timeout (default: 10s)
  request:
    streaming:
      first_byte: 30s          # How long to wait for the first chunk (default: 120s)
      chunk_interval: 10s      # Max silence between later chunks before treating as failure (default: 30s)
      total: 600s              # Total streaming budget (keep generous)

# 2. Health checks — detect backend failures proactively
health_checks:
  interval: 10s                # Check every 10s instead of 30s (default: 30s)
  timeout: 3s                  # Fail health checks faster (default: 5s)
  unhealthy_threshold: 2       # Mark unhealthy after 2 failures (default: 3)
  healthy_threshold: 1         # Recover after 1 success (default: 2)
  warmup_check_interval: 1s   # Fast checks during backend startup

# 3. Fallback chain — must be configured for mid-stream fallback to activate
fallback:
  enabled: true
  fallback_chains:
    "gpt-5.6-sol":
        - "gpt-5.6-terra"
        - "gpt-5.6-luna"
  fallback_policy:
    trigger_conditions:
      error_codes: [429, 500, 502, 503, 504]
      timeout: true
      connection_error: true
      backend_unhealthy: true

# 4. Mid-stream fallback — continuation mode (default: enabled)
streaming:
  mid_stream_fallback:
    enabled: true              # Use continuation mode (default)
    max_fallback_attempts: 3   # Allow more retries for resilience (default: 2)
    min_accumulated_tokens: 30 # Lower threshold for continuation vs restart (default: 50)

Parameter impact summary¶

Failure detection scenarios¶

Different failure types are detected at different speeds:

The most common scenario in production, a backend becoming unresponsive after it has started streaming, is governed by chunk_interval. A backend that accepts the request and then goes silent is governed by first_byte instead. Both are enforced on all three streaming paths (the mid-stream fallback relay, the Bedrock trait-stream pipeline, and the Gemini streaming pipeline). For latency-sensitive applications, lowering chunk_interval to 10–15 seconds is recommended, with model-specific overrides for slow models:

timeouts:
  request:
    streaming:
      chunk_interval: 10s      # Fast detection for most models
    model_overrides:
      gemini-2.5-pro:          # Reasoning models need longer intervals
        streaming:
          chunk_interval: 30s
          first_byte: 120s

Rate Limiting¶

The configurable, multi-dimension rate limiter is documented in Rate Limiting. Separately from it, Continuum Router has built-in, always-on protection for the model-listing endpoints, with fixed limits.

Built-in /v1/models Protection¶

GET /v1/models is rate-limited per client:

For this limiter, a client is identified by the first 16 characters of the Authorization: Bearer API key, falling back to the client IP address when no key is present. Each client has an independent quota.

POST /v1/models/refresh (force refresh) has a much tighter budget because each refresh clears the cache and fans out to every configured backend: 12 requests per minute with a burst cap of 3 per 5-second window. Only validated API keys get a per-key bucket; anonymous or invalid-token callers share a single global bucket, so spoofed headers cannot mint fresh quotas.

When a limit is exceeded, the endpoint returns 429 Too Many Requests with a message indicating whether the burst or the sustained limit was hit; rejections are also logged with the client identifier.

Cache TTL Optimization¶

To prevent cache poisoning during backend outages, empty model lists are cached for only 5 seconds initially (backing off up to 60 seconds for consecutive empty responses), while normal responses use the standard model-cache TTL.

Smart Routing¶

Smart routing classifies incoming requests by complexity and domain, then routes them to the most appropriate model tier using configurable policies. It combines a model tier registry (mapping models to tiers and domains) with a rule-based request classifier and a policy engine that maps classification results to routing decisions.

When model: "auto" is used in a chat completion request, the pipeline runs: classify the request, evaluate policies top-to-bottom, select a model from the matched tier. The same pipeline runs for all requests when intercept_all: true.

Configuration¶

smart_routing:
  enabled: true

  # Default tier when no profile matches and auto-inference is inconclusive.
  # 1 = Flagship, 2 = Standard, 3 = Lightweight. Defaults to 2.
  default_tier: 2

  # Model name that triggers smart routing. Defaults to "auto".
  virtual_model: "auto"

  # When true, all requests go through smart routing regardless of model name.
  intercept_all: false

  model_profiles:
    # Exact model name
    - model: "gpt-5.6-sol"
      tier: 1
      domains: [general, code, reasoning, creative]
      cost_per_1k_input_tokens: 0.005
      cost_per_1k_output_tokens: 0.030

    # Another exact match
    - model: "gpt-5.6-terra"
      tier: 2
      domains: [general, code]
      cost_per_1k_input_tokens: 0.0025
      cost_per_1k_output_tokens: 0.015

    # Glob pattern — matches all GGUF Q4_K_M quantized models
    - model_pattern: "*-q4_K_M"
      tier: 3
      domains: [general]

  # Routing policies: first match wins, top-to-bottom evaluation
  routing_policies:
    - name: "trivial_to_lightweight"
      when:
        complexity: [trivial, simple]
        domain: [general]
      route_to:
        tier: 3

    - name: "code_to_flagship"
      when:
        domain: [code]
        complexity: [moderate, complex, expert]
      route_to:
        tier: 1
        prefer_domains: [code]

    - name: "vision_required"
      when:
        requires: [vision]
      route_to:
        tier: 1
        require_capabilities: [vision]

    - name: "complex_to_flagship"
      when:
        complexity: [complex, expert]
      route_to:
        tier: 1

    - name: "default_to_standard"
      when: {}                      # Catch-all (always matches)
      route_to:
        tier: 2

Model Profile Cost Fields¶

Both cost_per_1k_input_tokens and cost_per_1k_output_tokens are USD per 1,000 tokens. Each is optional, and each must be finite and non-negative when present: 0 is valid and means the model is free, while a negative value, .nan, .inf, and -.inf are all rejected at config load and by PUT /admin/smart-routing/model-profiles.

The rejection is a hard error, not a warning. A config.yaml carrying such a value fails at startup with a message naming the offending profile, and a hot reload keeps the previous configuration.

The same validation call also reaches a pre-existing length limit for the first time: model and model_pattern must each be 200 characters or fewer. That limit previously applied only through the admin API; a config.yaml with an over-length model name or pattern now fails to load as well.

Only the input cost feeds model selection. The scorer awards up to 0.3 points on cost_per_1k_input_tokens, so a lower input cost wins among otherwise equal candidates in the same tier; a profile that omits the field scores a neutral 0.15 instead. cost_per_1k_output_tokens is informational: the admin API and the WebUI serve it back, and it is validated identically, but no routing decision reads it.

Tier Classification¶

Domain Specialization Tags¶

Auto-Inference¶

When a model has no matching explicit profile or glob pattern, the router infers its tier automatically using three sources in priority order:

1. Pricing (from model-metadata.yaml, where pricing is USD per 1M tokens): input cost >= $3/1M tokens maps to Flagship; >= $0.50/1M to Standard; below that to Lightweight. Zero-cost models skip pricing inference.

2. Capabilities (from model-metadata.yaml): models with 3+ high-value capabilities (vision, reasoning, audio, video, function_calling, tool) map to Flagship; 1+ such capability or 3+ total capabilities map to Standard.

3. Name heuristics: keywords like pro, ultra, opus, sonnet, turbo map to Flagship; keywords like mini, small, tiny, nano, lite, flash, haiku and quantization markers (q4_, q5_, q8_, gguf, gptq, awq) map to Lightweight.

Auto-inferred results are cached per model ID (up to 10,000 entries). The cache clears on hot-reload and when the /admin/smart-routing/model-profiles PUT endpoint is called.

Glob Pattern Syntax¶

Patterns use * as the only wildcard character. Multiple wildcards are supported.

Request Classifier¶

The rule-based classifier analyzes each request using 11 signal types and produces a ClassificationResult containing complexity level, domain tag, required capabilities, and confidence score.

Complexity Levels¶

Classification Signals¶

Each detected signal contributes to the final complexity level and domain tag. Conflicting signals (e.g., both creative and code markers present) reduce the confidence score.

The domain is decided by the strongest intent signal (code_blocks, inline_code, math_notation, creative_markers, analysis_markers, code_intent). multilingual is a script observation rather than an intent, so it decides the domain only when no intent signal fired at all.

code_intent is what recognizes a request to write code that contains none. Before it existed the code domain was keyed entirely on literal markup, so "Implement the rate limiting logic for this REST API." carried no backticks, fired no code signal, and fell through to general or multilingual, taking every domain: [code] policy with it. The signal needs two distinct entries from the code keyword list to match, and that threshold is the design rather than a tuning knob: one marker alone is either an action or an object, and each fails on its own, since bare actions would put "Write a poem about autumn leaves" in the code domain and bare objects would put "What is a REST API?" there. It is skipped outright when the request already carries a fenced block or inline backticks, so a request with real code markup keeps both its domain and its confidence unchanged. Its strength of 0.75 sits below math_notation (0.8), so a LaTeX-heavy request stays reasoning, and above multilingual (0.7), so an unmarked Korean request stops collapsing into the script observation.

Language Awareness¶

The classifier detects the language of the user's own message text and publishes it on the classification result. Detection is script-based: Hangul means ko, kana means ja, Han without kana means zh, Latin script is reported as en, and anything else is und (undetermined). A tenth of the letters is enough for Hangul or kana, so a Korean or Japanese request that quotes English technical vocabulary is still detected as Korean or Japanese.

The detected language selects the keyword table the complexity, creative, and analysis signals match against. English and Korean ship built in, and every table includes the English keywords, so a request that mixes languages keeps firing the signals it would have fired on the English list alone. A language with no table of its own falls back to English.

Request length is estimated from characters, weighted by script (a Hangul syllable or a CJK ideograph counts as two Latin characters), not from UTF-8 bytes. A Korean request and its English translation therefore land in the same complexity band.

Primary language¶

classifier.rule.primary_language names the language this deployment's traffic is normally written in, as a short BCP-47 tag (ko, ko-KR, ja, fr; case-insensitive, only the primary subtag is read).

Setting it suppresses the multilingual signal for requests in that language. The signal fires on any text that is mostly non-ASCII, so on a Korean deployment it fired on every ordinary Korean request, and a Korean greeting (which carries no intent signal at all) was classified multilingual where its English translation was classified general. Naming the primary language says that text in that language is not itself a translation task, and such a request is then classified on its intent alone. Requests in any other language keep the signal.

It also selects the keyword table for Latin-script text, since detection reports every Latin-script language as en. primary_language: fr is what makes a keywords entry for fr reachable. A tag detection recognizes on its own (ko, ja, zh) never overrides detection: Latin-script text on a Korean deployment is still matched against the English table.

The setting is optional. Left unset, every language keeps the multilingual signal exactly as before, so an existing configuration's domain assignments do not change.

Custom keywords¶

classifier.rule.keywords adds keywords per language. Entries are added to the built-in tables rather than replacing them, every language inherits the English built-ins, and values are trimmed and lowercased with duplicates dropped. A tag with no shipped table (fr, de) becomes a new table on top of the English base, reached by setting primary_language to the same tag.

Matching works the same way for a custom keyword as for a built-in one. A single-word keyword is a plain substring match, so the Korean stem 설명 matches inside 설명해줘 and poem matches inside poems. A multi-word keyword matches when its words appear in order within one sentence with at most two tokens between consecutive words, which is what lets write a story fire on "write a short story" and 시를 써 fire on "시를 하나 써줘" without the list having to spell out every modifier and particle placement. A sentence terminator (., !, ?, a newline) closes that window, so words on opposite sides of one never form a match however close together they sit.

Each word of a multi-word keyword has to line up with a whole token, ignoring punctuation at the token edges. The single exception is the last word, which may be followed by more characters inside the same token when it ends in a non-Latin letter. That is what lets the Korean verb stem 써 match 써줘. It is confined to the last word so a short leading noun cannot latch onto an unrelated one: the 시 in 시 한 편 must not match 시스템. Latin-script words never take the exception at all, because a one- or two-letter prefix would match a large part of the dictionary.

One class of marker is held to adjacency: if both the first and the last word are Latin function words (a, the, is, what, once, upon, and the like), the gap rule is switched off for it. Such a marker is an idiom rather than a phrase built around a head word, and its meaning lives in the adjacency, so once upon would otherwise fire on "Once agreed upon, the schema is frozen" and what is on "What throughput is achievable here". Only the outer words are consulted, so write a story keeps its gap tolerance through write and story. The rule reads Latin script only: Korean marks the same grammatical roles with endings attached to a word rather than with separate words, so no Korean marker is affected, and a custom idiom in another Latin-script language gets gap tolerance with no equivalent protection.

Gap tolerance cannot tell a modifier from a compound noun, and that is a real cost, not a rough edge. "Write a short story" and "Write a user story" have the same shape, so the second now classifies as creative even though it names a requirements artifact. If your traffic carries phrases like that and the misrouting matters, keep the affected marker out of keywords and match on something narrower.

smart_routing:
  classifier:
    rule:
      # The language this deployment's traffic is written in. Suppresses the
      # `multilingual` domain signal for that language. Optional.
      primary_language: ko

      # Extra keywords, merged into the built-in tables. Optional. A `fr`
      # entry here would need `primary_language: fr` to be reachable.
      keywords:
        ko:
          complex: ["커널 스케줄러"]
          creative: ["판소리 사설"]
          # `code` fires only when two distinct entries match, so add
          # entries in pairs: an action and an object.
          code: ["배포", "헬름 차트"]

Both fields hot-reload: a change rebuilds the rule classifier in place.

LLM-Based Classifier¶

The rule-based classifier is fast but sometimes ambiguous. When that is not accurate enough, the LLM-based classifier sends the request to a small, cheap model for classification. Three operating modes are supported via classifier.method:

Hybrid mode is the recommended production setting: it adds latency only for genuinely ambiguous requests (typically 10-20% of traffic), while trivial and clear-cut requests are classified in microseconds.

smart_routing:
  enabled: true

  classifier:
    # Classification method: "rule" (default), "llm", or "hybrid".
    method: hybrid

    rule:
      # Confidence below this threshold triggers LLM escalation in hybrid mode.
      # Range: 0.0 – 1.0. Default: 0.7.
      confidence_threshold: 0.7

      # Language this deployment's traffic is written in, as a short BCP-47
      # tag. Suppresses the `multilingual` domain signal for that language.
      # Optional; see "Language Awareness" above.
      primary_language: ko

      # Extra classification keywords per language, added to the built-in
      # tables. Optional.
      keywords:
        ko:
          creative: ["판소리 사설"]
          # `code` fires only when two distinct entries match, so add
          # entries in pairs: an action and an object.
          code: ["배포", "헬름 차트"]

    llm:
      # Model used for classification. Any fast, cheap model works well.
      model: "gpt-5.6-terra"

      # Backend name to route classification requests to. Must be a configured
      # backend. If omitted, the router uses the first backend that ordinary
      # user traffic can reach, skipping any `internal: true` or
      # `enabled: false` entry. See "Classifier Backend Selection" below.
      backend: "openai-fast"

      # Maximum time allowed for a classification request (milliseconds).
      timeout_ms: 2000

      # Maximum input tokens sent to the classifier (content is truncated).
      max_input_tokens: 500

      # Number of retries after a parse failure (0 or 1). Default: 1.
      max_retries: 1

      # Temperature for classification. 0.0 gives deterministic output.
      # Range: 0.0 – 2.0. Default: 0.0.
      temperature: 0.0

      # Maximum output tokens in the classification response.
      max_output_tokens: 150

      # Structured output strategy. "auto" selects the best method for the
      # configured backend: json_schema (OpenAI/vLLM), tool_use (Anthropic),
      # json_object (Ollama/Gemini/LM Studio), prompt_only (others).
      structured_output: auto   # auto | json_schema | tool_use | json_object | prompt_only

      # Include built-in few-shot examples in the system prompt.
      few_shot_examples: true

      # Custom few-shot examples appended after the built-in ones.
      custom_examples:
        - user: "What is the recommended dose of ibuprofen?"
          classification:
            complexity: simple
            domain: medical

      # Cache TTL for classification results (seconds). Default: 300.
      cache_ttl_seconds: 300

      # Maximum number of cached entries. Default: 10000.
      max_cache_entries: 10000

      # Disable the LLM classifier when load reaches this state.
      # "critical" (default) or "warning". Set to "" to never disable.
      disable_under_load_state: critical

    # Extend the built-in complexity taxonomy with custom levels.
    custom_complexity_levels:
      - name: specialized
        description: "Requires a domain-specific fine-tuned model"
        rank: 6   # Optional ordering hint (higher = harder)

    # Extend the built-in domain taxonomy with custom categories.
    custom_domains:
      - name: medical
        description: "Medical and clinical questions"

rule.confidence_threshold and llm.temperature are each validated at config load: confidence_threshold is bounded to [0.0, 1.0] inclusive, and temperature to [0.0, 2.0] inclusive, matching the range the router already enforces for the canonical temperature request parameter elsewhere in this document. Both bounds are inclusive, so 0.0 and the upper bound are legitimate settings, not accidents: confidence_threshold: 0.0 makes hybrid mode always escalate to the LLM classifier, and confidence_threshold: 1.0 makes it never escalate. An out-of-range or non-finite value (.nan, .inf, -.inf) is rejected the same way the model profile cost fields above are: a config.yaml carrying such a value fails at startup with a message naming the field, and a hot reload keeps the previously loaded configuration.

When classifier.backend (or the classifier's resolved backend URL) points at a native Anthropic backend and classifier.llm.model is a Claude Opus or Sonnet 4.7 or later, the configured temperature is silently dropped from the classification request rather than sent and rejected, the same sampling-parameter restriction described for the main request path in Anthropic Extended Thinking Models.

The classifier dispatches its classification request through the same backend implementation that serves normal traffic, so every configured backend type is supported and uses that backend's own authentication and request transformation. If classifier.llm.backend names a backend that is not in the live backend pool, classification fails for that call with an error naming the backend and falls back to the rule-based classifier, rather than failing at startup.

Classifier Backend Selection¶

Classification sends the user's own prompt text to the classifier backend, so which backend gets picked decides where that content goes. The choice is made once, when the router builds its state and again on each hot reload, not per request, so a given configuration keeps sending classification prompts to the same backend until the configuration changes.

classifier.llm.backend and its absence are resolved differently on purpose:

Earlier releases resolved the omitted case to literally the first entry in backends, with no visibility check, so an internal: true guard backend or a powered-off enabled: false standby in first position received every classifiable request's prompt text. A deployment with no hidden backends resolves exactly as it did before.

Structured Output Strategies¶

The LLM classifier needs structured JSON from the classifier model. The auto strategy picks the right mechanism based on the backend type, but you can override it explicitly:

When the classifier response cannot be parsed, the router retries once with a correction prompt (controlled by max_retries). If the retry also fails, the result from the rule-based classifier is used instead.

Classification Cache¶

Classification results are cached in memory with a configurable TTL to avoid repeated LLM calls for the same request. The cache key is a SHA-256 hash of the truncated user message, so semantically identical requests share the same cached result. The cache is per-process and not shared across router instances.

Custom Taxonomy¶

Both complexity levels and domain tags are extensible. Custom values added via custom_complexity_levels and custom_domains appear in the classifier's system prompt and are accepted in the structured-output schema. Routing policies can reference custom values just like built-in ones:

routing_policies:
  - name: specialized_to_flagship
    when:
      complexity: [specialized]
    route_to:
      tier: 1

Bypass Header¶

The LLM classifier sends an X-Smart-Route-Bypass: true header with its classification requests. The router skips smart routing for any request carrying this header, preventing circular classification loops when the classifier backend is itself behind the same router instance.

Routing Policies¶

Policies are evaluated top-to-bottom; the first match wins. If no policy matches and no catch-all is defined, the request falls back to default_tier.

Policy Condition Logic¶

• Fields within a when block are AND-ed: all specified fields must match.
• Values within a single field are OR-ed: complexity: [trivial, simple] matches either.
• when: {} is a catch-all that always matches. A condition that sets only an identity field (key_tier, org, language, header) is not a catch-all.

Policy Fields¶

when conditions:

Identity Conditions¶

The first three fields describe what the request is. The last four describe who sent it and how they asked for it to be handled, so one auto alias can route differently per key, per organization, per request language, or on an explicit client hint.

routing_policies:
  # A paying tier gets the flagship models.
  - name: "build_tier_flagship"
    when:
      key_tier: [tier_build]
    route_to:
      tier: 1

  # One organization is pinned to the cheapest tier regardless of complexity.
  - name: "sandbox_org_lightweight"
    when:
      org: [org_sandbox]
    route_to:
      tier: 3

  # Korean requests go to a model that handles them well.
  - name: "korean_to_flagship"
    when:
      language: [ko]
    route_to:
      tier: 1

  # A client can ask for a more thorough answer.
  - name: "thorough_hint"
    when:
      header:
        x-route-hint: [thorough]
    route_to:
      tier: 1

  - name: "default_to_standard"
    when: {}
    route_to:
      tier: 2

key_tier matches the Continuum Hub tier id (tier_id) resolved for the request's API key. A binary built without the control-plane feature never resolves a tier, and neither does a request whose key is not hub-bound, so a policy naming key_tier simply never matches there. It never matches everything: a tier-gated policy fails closed rather than opening up where tiers do not exist. continuum-router config validate warns when a policy uses key_tier on a build that cannot resolve one.

org matches the organization_id of the authenticated API key. An unauthenticated request carries no organization and never matches.

language reads the language the classifier detected in the user's own message text, the same detection that drives the language-aware keyword tables. Only the primary subtag is significant and the comparison is case-insensitive, so ko, KO, and ko-KR all name Korean. Two absence cases differ deliberately: a request with no letters at all (an empty message, or digits and punctuation only) carries no language and matches no language clause, while text that was examined but could not be attributed carries und and matches a clause that lists und.

header reads request headers. Names and values are both matched case-insensitively, and an entry with an empty value list is a presence check (x-route-hint: [] matches the header with any value). Values are bounded at 256 bytes; a longer value is treated as absent rather than truncated, so a client cannot match a policy by sending a long value that merely starts with the configured one. Because a client sets these headers itself, use header to express a caller's preference and gate anything privileged on key_tier or org, which a client cannot set. Header values never appear in a metric label or a log line.

When smart_routing.debug_headers is on, x-smart-route-policy names the policy that matched, which is the quickest way to confirm an identity condition is firing. POST /admin/smart-routing/simulate also accepts key_tier, org, and headers fields alongside payload, so an operator can check a tier-gated policy without holding a key on that tier.

route_to action:

Model Selection Within a Tier¶

When multiple models belong to the matched tier, the selector scores them using:

1. Domain preference match (soft bonus)
2. Capability match (hard filter if require_capabilities is set)
3. Cost scoring (lower cost scores higher within a tier)
4. Random tiebreak for equal-score models

If no models are available in the matched tier, the router tries adjacent tiers in order (e.g., if Lightweight is empty, tries Standard, then Flagship).

Load-Aware Dynamic Tier Adjustment¶

Under normal conditions, smart routing picks the tier that best matches each request. When traffic spikes or latency rises, the load monitor tracks real-time metrics and automatically shifts routing to lighter model tiers until the system recovers.

Load management is disabled by default. Enable it under smart_routing.load_management:

smart_routing:
  enabled: true
  load_management:
    enabled: true

    # How often load metrics are evaluated (milliseconds). Default: 1000.
    assessment_interval_ms: 1000

    # Thresholds for entering Warning and Critical states.
    # Any single threshold being exceeded triggers the state transition.
    thresholds:
      warning:
        requests_per_second: 100
        avg_latency_ms: 3000
        error_rate: 0.05        # Range: 0.0 – 1.0
        in_flight_requests: 50
        kv_cache_usage: 0.7     # Engine-side. Range: 0.0 – 1.0
        waiting_requests: 4     # Engine-side queue depth
      critical:
        requests_per_second: 200
        avg_latency_ms: 5000
        error_rate: 0.15
        in_flight_requests: 100
        kv_cache_usage: 0.9
        waiting_requests: 16

    # Routing restrictions applied per load state.
    degradation:
      warning:
        max_tier: 2           # Cap routing at Standard tier
        prefer_quantized: false
        reject_expert: false
      critical:
        max_tier: 3           # Cap routing at Lightweight tier
        prefer_quantized: true
        reject_expert: false

    # Recovery behavior.
    recovery:
      cooldown_seconds: 30    # Minimum time before downgrading the load state
      hysteresis_factor: 0.8  # Metric must drop to 80% of threshold to recover. Range: 0.0 – 1.0

Engine-Side Inputs¶

requests_per_second, avg_latency_ms, error_rate, and in_flight_requests are router-local: they are computed from the router's own request counters. kv_cache_usage and waiting_requests are engine-side, read from the engine statistics the router already polls (engine_stats), and they exist because a GPU can be saturated while every router-local number still looks healthy. A backend whose KV cache is nearly full still accepts connections and still passes its health probe, and router latency only moves once the queue has already built, which is too late to shift traffic to a lighter tier.

Both fields read the highest value across the backends whose engine-stats snapshot is currently fresh. Averaging would let three idle backends hide one saturated one, which is the incident the thresholds exist for.

Freshness is not a separate setting. The monitor reads the same TTL-bounded view the engine-load routing term uses, so a snapshot counts only while it is younger than engine_stats.interval * routing.engine_load.max_staleness_intervals. Three consequences follow:

• A deployment that does not enable engine_stats has no snapshot at all, so the engine thresholds never fire regardless of what they are set to.
• A stalled or failing poller degrades the engine thresholds to silence rather than pinning the router to the last value it saw.
• An engine that does not report one of the two fields makes no statement about it. An absent field is never read as zero.

Each field is opt-in on its own: leave kv_cache_usage or waiting_requests unset and it is not consulted, exactly like the router-local thresholds. kv_cache_usage is bounded to [0.0, 1.0] and validated at config load alongside error_rate and hysteresis_factor.

Setting either field inside a tier_thresholds override is accepted but behaves identically to the global threshold, because engine saturation is a property of the backend pool rather than of a routing tier. This matches how requests_per_second, avg_latency_ms, and error_rate already behave in a tier override; in_flight_requests is the only genuinely tier-scoped field.

Load States¶

When a request would normally route to Flagship (tier 1) but the load state is Warning, it is silently downgraded to Standard. The routing decision log records the adjusted policy name as <original_policy>__load_warning or <original_policy>__load_critical.

Hysteresis and Cooldown¶

Rapid oscillation between load states can itself cause instability. Two mechanisms prevent it:

• Hysteresis: to leave Warning state, a metric must drop below threshold * hysteresis_factor (default 0.8), not just below the threshold. A system that entered Warning at 100 RPS stays there until RPS drops below 80.
• Cooldown: after any state transition, recovery to a lower state is blocked for cooldown_seconds (default 30). Escalation (Normal to Warning, Warning to Critical) always bypasses the cooldown.

recovery.hysteresis_factor and every error_rate field, in thresholds.warning, thresholds.critical, and any tier_thresholds override, are bounded to [0.0, 1.0] inclusive and validated at config load. An out-of-range or non-finite value (.nan, .inf, -.inf) fails startup with a message naming the field, and for a tier_thresholds entry, naming the tier key as well; a hot reload carrying the same value keeps the previously loaded configuration instead of applying it. 0.0 stays inside the valid range for hysteresis_factor, and it is worth calling out on its own: threshold * 0.0 is always 0.0, so any non-zero traffic keeps the stay-in-Warning-or-Critical check true, and the router remains in an elevated load state for as long as any traffic flows at all, returning to Normal only once a snapshot is completely idle.

Per-Tier Threshold Overrides¶

If different tiers have different capacity characteristics, you can override thresholds per tier:

smart_routing:
  load_management:
    enabled: true
    thresholds:
      warning:
        requests_per_second: 100
    tier_thresholds:
      "1":                      # Tier 1 (Flagship) has a lower RPS tolerance
        warning:
          requests_per_second: 50
      "3":                      # Tier 3 (Lightweight) can handle more
        warning:
          requests_per_second: 300

Prometheus Metrics¶

When the metrics feature is enabled, the following counters and gauges are exported for load management:

reason names the rule that drove the transition and is one of four fixed values: router_metrics (a router-local global threshold), engine_stats (an engine-side global threshold), tier_threshold (any tier_thresholds override, whichever signal it was written against, because that names the block to edit), and recovery (no threshold held any more). The set is closed, so the series count stays bounded no matter how many backends or models are configured.

The LLM classifier exports six additional metrics:

Debug Response Headers¶

Set debug_headers: true to include smart routing decision details in HTTP response headers. This is intended for development and staging environments.

smart_routing:
  enabled: true
  debug_headers: true  # Enable in development/staging

When enabled, smart-routed responses include:

Admin API¶

Smart routing exposes several admin endpoints under /admin/smart-routing/ for observability and management. The full endpoint reference is in the Admin API documentation.

Key endpoints:

• GET /status -- overall status, load state, policy count
• POST /classify -- classify a request without routing (diagnostic)
• POST /simulate -- simulate the full routing pipeline
• GET /policies and PUT /policies -- view and hot-reload policies
• GET /load-state -- current load state with assessment details. The snapshot includes an engine object (kv_cache_usage, waiting_requests, fresh_backends) carrying the engine-side numbers the assessment was made on; the first two are null when no fresh snapshot stated them, which is not the same as an idle 0, and fresh_backends says how many backends contributed
• GET /cache/stats and POST /cache/clear -- LLM classifier cache management

Structured Logging¶

All smart routing decisions are logged at DEBUG level with structured fields:

level=DEBUG msg="Smart routing decision"
  source_model="auto"
  target_model="gpt-5.6-terra"
  complexity="simple"
  domain="general"
  policy="trivial_to_lightweight"
  load_state="normal"
  classifier="rule_based"
  confidence=0.92
  classification_ms=0.3

Load state transitions and policy changes are logged at INFO level.

Hot Reload¶

The smart_routing section reloads immediately when the config file changes. After a reload, the inferred-profile cache is cleared so all models are re-evaluated on the next request. Routing policies in routing_policies and load_management settings also take effect immediately without restarting the server. Policies can also be updated at runtime via the PUT /admin/smart-routing/policies endpoint.

Environment-Specific Configurations¶

Development Configuration¶

# config/development.yaml
server:
  bind_address: "127.0.0.1:8080"

backends:
  - name: "local-ollama"
    url: "http://localhost:11434"

health_checks:
  interval: "10s"                 # More frequent checks
  timeout: "5s"
  unhealthy_threshold: 3
  healthy_threshold: 2
  endpoint: "/health"

logging:
  level: "debug"                  # Verbose logging
  format: "pretty"                # Human-readable

Production Configuration¶

# config/production.yaml
server:
  bind_address: "0.0.0.0:8080"
  workers: 8                      # More workers for production
  connection_pool_size: 300       # Larger connection pool

backends:
  - name: "primary-openai"
    url: "https://api.openai.com"
    weight: 3
  - name: "secondary-azure"
    url: "https://azure-openai.example.com"
    weight: 2
  - name: "fallback-local"
    url: "http://internal-llm:11434"
    weight: 1

health_checks:
  interval: "60s"                 # Less frequent checks
  timeout: "15s"                  # Longer timeout for network latency
  unhealthy_threshold: 5          # More tolerance
  healthy_threshold: 3
  endpoint: "/health"

timeouts:
  connection: "10s"
  request:
    standard:
      first_byte: "30s"           # Deprecated and inert; `total` is the budget
      total: "120s"               # Bounded timeout for production
    streaming:
      first_byte: "120s"          # Enforced deadline for the first SSE chunk
      chunk_interval: "30s"
      total: "600s"

retry:
  max_attempts: 3                 # Bounded retries
  initial_delay: "100ms"
  max_delay: "10s"
  backoff_multiplier: 2.0
  jitter: true
  retryable_status_codes: [429, 502, 503, 504]
  retryable_errors: [ConnectionError, TimeoutError]
  timeout: "30s"

logging:
  level: "warn"                   # Less verbose logging
  format: "json"                  # Structured logging

Container Configuration¶

# config/container.yaml - optimized for containers
server:
  bind_address: "0.0.0.0:8080"
  workers: 0                      # Auto-detect based on container limits

backends:
  - name: "backend-1"
    url: "${BACKEND_1_URL}"       # Environment variable substitution
  - name: "backend-2"
    url: "${BACKEND_2_URL}"

health_checks:
  interval: "30s"
  timeout: "5s"
  unhealthy_threshold: 3
  healthy_threshold: 2
  endpoint: "/health"

logging:
  level: "info"                   # Render a different value before validation if needed
  format: "json"                  # Always JSON in containers