Advanced Configuration¶

Global Prompts¶

Global prompts allow you to inject system prompts into all requests, providing centralized policy management for security, compliance, and behavioral guidelines. Prompts can be defined inline or loaded from external Markdown files.

Basic Configuration¶

global_prompts:
  # Inline default prompt
  default: |
    You must follow company security policies.
    Never reveal internal system details.
    Be helpful and professional.

  # Merge strategy: prepend (default), append, or replace
  merge_strategy: prepend

  # Custom separator between global and user prompts
  separator: "\n\n---\n\n"

External Prompt Files¶

For complex prompts, you can load content from external Markdown files. This provides:

• Better editing experience with syntax highlighting
• Version control without config file noise
• Hot-reload support for prompt updates

global_prompts:
  # Directory containing prompt files (relative to config directory)
  prompts_dir: "./prompts"

  # Load default prompt from file
  default_file: "system.md"

  # Backend-specific prompts from files
  backends:
    anthropic:
      prompt_file: "anthropic-system.md"
    openai:
      prompt_file: "openai-system.md"

  # Model-specific prompts from files
  models:
    gpt-5.6-sol:
      prompt_file: "gpt5-6-sol-system.md"
    claude-opus-5:
      prompt_file: "claude-opus-system.md"

  merge_strategy: prepend

Prompt Resolution Priority¶

When determining which prompt to use for a request:

1. Model-specific prompt (highest priority) - global_prompts.models.<model-id>
2. Backend-specific prompt - global_prompts.backends.<backend-name>
3. Default prompt - global_prompts.default or global_prompts.default_file

For each level, if both prompt (inline) and prompt_file are specified, prompt_file takes precedence.

Merge Strategies¶

REST API Management¶

Prompt files can be managed at runtime via the Admin API:

# List all prompts
curl http://localhost:8080/admin/config/prompts

# Get specific prompt file
curl http://localhost:8080/admin/config/prompts/prompts/system.md

# Update prompt file
curl -X PUT http://localhost:8080/admin/config/prompts/prompts/system.md \
  -H "Content-Type: application/json" \
  -d '{"content": "# Updated System Prompt\n\nNew content here."}'

# Reload all prompt files from disk
curl -X POST http://localhost:8080/admin/config/prompts/reload

See Admin REST API Reference for complete API documentation.

Security Considerations¶

• Path Traversal Protection: All file paths are validated to prevent directory traversal attacks
• File Size Limits: Individual files limited to 1MB, total cache limited to 50MB
• Relative Paths Only: Prompt files must be within the configured prompts_dir or config directory
• Sandboxed Access: Files outside the allowed directory are rejected

Hot Reload¶

Global prompts support immediate hot-reload. Changes to prompt configuration or files take effect on the next request without server restart.

Model Metadata¶

Continuum Router supports rich model metadata to provide detailed information about model capabilities, pricing, and limits. This metadata is returned in /v1/models API responses and can be used by clients to make informed model selection decisions.

Metadata Sources¶

Model metadata can be configured in three ways (in priority order):

1. Backend-specific model_configs (highest priority)
2. External metadata file (model-metadata.yaml)
3. No metadata (models work without metadata)

External Metadata File¶

Create a model-metadata.yaml file:

models:
  - id: "gpt-5.6-sol"
    aliases:                    # Alternative IDs that share this metadata
      - "gpt-5.6"
    metadata:
      display_name: "GPT-5.6 Sol"
      summary: "Frontier GPT-5.6 model for complex professional work"
      capabilities: ["chat", "vision", "code", "reasoning", "tool"]
      knowledge_cutoff: "2026-02"
      pricing:
        input_tokens: 5.0    # USD per 1M tokens
        output_tokens: 30.0  # USD per 1M tokens
      limits:
        context_window: 1050000
        max_output: 128000

  - id: "llama-3-70b"
    aliases:                    # Different quantizations of the same model
      - "llama-3-70b-instruct"
      - "llama-3-70b-chat"
      - "llama-3-70b-q4"
      - "llama-3-70b-q8"
    metadata:
      display_name: "Llama 3 70B"
      summary: "Open-source model with strong performance"
      capabilities: ["text", "code"]
      knowledge_cutoff: "2023-12"
      pricing:
        input_tokens: 1.0    # USD per 1M tokens
        output_tokens: 2.0   # USD per 1M tokens
      limits:
        context_window: 8192
        max_output: 2048

Image Model Pricing¶

Image models price on axes that per-token rates cannot express, so pricing accepts three further optional keys:

  # Billed per generated image: flat by size, flat by named tier, or
  # size then quality. All three shapes are supported.
  - id: "dall-e-2"
    metadata:
      pricing:
        input_tokens: 0        # literal, not missing data
        output_tokens: 0
        per_image:             # USD per generated image
          1024x1024: 0.02
          512x512: 0.018

  - id: "dall-e-3"
    metadata:
      pricing:
        input_tokens: 0
        output_tokens: 0
        per_image:
          1024x1024: { standard: 0.04, hd: 0.08 }
      limits:
        context_window: 0            # 0 = not applicable (no token context)
        max_output: 0
        max_prompt_length: 4000      # characters, not tokens
        supported_sizes: ["1024x1024", "1792x1024", "1024x1792"]
        max_n: 1                     # max images per request
        supported_qualities: ["standard", "hd"]
        supported_styles: ["vivid", "natural"]

  # Billed per token, with image input at its own rate alongside text input.
  - id: "gpt-image-2"
    metadata:
      pricing:
        input_tokens: 5.0
        output_tokens: 30.0
        image_input_tokens: 8.0          # USD per 1M image input tokens
        cached_image_input_tokens: 2.0   # USD per 1M cached image input tokens

• image_input_tokens and cached_image_input_tokens: Billed per 1M tokens in USD, using the same unit as input_tokens. These rates add to the text token rates rather than replacing them. Note that cached_image_input_tokens is an explicit USD price, whereas cached_input_discount is a discount ratio (0.0 to 1.0).
• per_image: Billed in USD per generated image. Models using per-image pricing set input_tokens: 0 and output_tokens: 0 intentionally to indicate that token pricing does not apply.

All three keys are optional, additive, and exposed on /v1/models/extended and /v1/models/{model} (see Pricing Object Fields).

Long-Context Pricing¶

Models that charge a higher rate after a prompt threshold can declare long_context. The tier applies to the whole request only when the prompt token count is greater than threshold_tokens; an equal count uses the base rates.

  - id: "claude-haiku-5-5"
    metadata:
      pricing:
        input_tokens: 0.1
        output_tokens: 0.5
        long_context:
          threshold_tokens: 100000
          input_tokens: 0.5
          output_tokens: 2.5

The threshold must be between 1 and 10,000,000 tokens. Both tier rates are non-negative finite values in USD per 1M tokens. The field is exposed through the model APIs; smart-routing tier inference continues to use the base input rate.

Image models can also declare optional limits fields, as shown in the dall-e-3 example above: max_prompt_length (in characters, not tokens), supported_sizes, max_n, supported_qualities, supported_output_formats, supported_styles, and supports_streaming. These fields are informational for API consumers; the router enforces image request limits per model family in code rather than reading them from metadata. Setting context_window: 0 and max_output: 0 indicates that token context bounds do not apply. Unknown keys in a metadata file are ignored with a load-time warning that logs the dotted field path.

Both pricing and limits values are range-checked (for example, context_window and max_output accept 0 to 10,000,000):

• Runtime load: An out-of-range value logs a warning naming the model and field, and the router continues serving the configured value as written.
• metadata download: An out-of-range value is treated as a fatal error and aborts the download before modifying any file on disk, preventing a malformed upstream file from replacing a working configuration.

Reference it in your config:

model_metadata_file: "model-metadata.yaml"

model_metadata_file is only the base layer. The router also assembles metadata from model-metadata.d/ drop-in directories and a model_metadata_dirs config key, so an operator's overrides do not have to live in the file metadata download overwrites. See Layered model metadata for the search order, the merge rules, and metadata show.

Thinking Pattern Configuration¶

Some models output reasoning/thinking content in non-standard ways. The router supports configuring thinking patterns per model to properly transform streaming responses.

Pattern Types:

Configuration Example:

models:
  - id: nemotron-3-nano
    metadata:
      display_name: "Nemotron 3 Nano"
      capabilities: ["chat", "reasoning"]
      # Thinking pattern configuration
      thinking:
        pattern: unterminated_start
        end_marker: "</think>"
        assume_reasoning_first: true

Thinking Pattern Fields:

How It Works:

When a model has a thinking pattern configured:

1. Streaming responses are intercepted and transformed
2. Content before end_marker is sent as reasoning_content field
3. Content after end_marker is sent as content field
4. The output follows OpenAI's reasoning_content format for compatibility

Example Output:

// Reasoning content (before end marker)
{"choices": [{"delta": {"reasoning_content": "Let me analyze..."}}]}

// Regular content (after end marker)
{"choices": [{"delta": {"content": "The answer is 42."}}]}

Buffered classification¶

assume_reasoning_first has to guess before it has evidence. It sends the leading tokens out as reasoning_content and waits for </think> to switch channels. A hybrid model that skips its thinking phase and answers directly never emits that marker, so the whole answer arrives on the reasoning channel and clients that render only content show an empty response.

Setting buffered: true replaces the guess with a decision. The router withholds the leading tokens and classifies them from what actually arrives:

models:
  - id: nemotron-3-nano
    metadata:
      thinking:
        pattern: unterminated_start
        end_marker: "</think>"
        assume_reasoning_first: true
        buffered: true
        max_buffer_size: 51200
        reasoning_timeout: "10s"

The decision window starts at the first token, not at request time, and a stream that goes quiet inside the window still resolves on time: the router wakes on the deadline rather than waiting for the next chunk.

buffered is only honored where the blind emission it replaces applies, which is pattern: unterminated_start with assume_reasoning_first: true. On any other pattern it is ignored. It is off by default, so existing deployments keep their current behavior until they opt in.

Time to first token:

Buffering trades TTFT for correct classification. Nothing reaches the client until the decision resolves, so first-token latency becomes the length of the model's thinking phase, bounded by reasoning_timeout. Measured against a mock backend with a 400 ms thinking phase (tests/streaming/thinking_buffered_streaming_test.rs):

Streams that never enter the decision window pay nothing: with buffered: false, or on any other pattern, the transformer arms no timer and holds no extra buffer. Keep reasoning_timeout at or below the latency your clients tolerate on an empty screen, and prefer buffered mode for hybrid models where a direct answer is common.

Responses-API-only Models¶

OpenAI exposes some models exclusively via the Responses API (/v1/responses). These models are not reachable through /v1/chat/completions, so a request that targets them on a Chat Completions endpoint returns a 404 not_found from upstream.

The responses_only capability flag marks such models so the router can dispatch them to the Responses API surface instead. The flag defaults to false, so existing model entries do not need to be touched.

Configuration Example:

models:
    - id: gpt-5.4-pro
      metadata:
        display_name: "GPT-5.4 Pro"
        capabilities: ["chat", "vision", "code", "reasoning", "tool"]
        # Served only on /v1/responses; not available on /v1/chat/completions.
        responses_only: true
        limits:
          context_window: 1050000
          max_output: 128000

Models marked Responses-API-only out of the box¶

The list below is kept in sync with model-metadata.yaml and the built-in OpenAI registry (src/infrastructure/backends/openai/models/gpt5_family.rs). When a new Responses-API-only model is added upstream, both files should be updated together.

The flag follows the same lookup priority chain as the rest of the metadata (backend model_configs > model-metadata.yaml > built-in OpenAI metadata), so an operator-supplied entry can override the default for any model.

Marking a new model as Responses-API-only¶

To mark an additional model as Responses-API-only, add responses_only: true to the model entry's metadata block in any of the supported sources. Use the lookup priority that fits the deployment scope:

• model-metadata.yaml for a router-wide default that applies to every backend. Add the flag alongside the existing capability metadata; no other field needs to change. This is the recommended location for newly-released Pro models that are uniformly Responses-API-only across providers.
• Backend model_configs in config.yaml for a backend-specific override (for example, when a self-hosted clone of a Pro model is exposed on a Chat Completions endpoint and should not be dispatched to /v1/responses). A backend-level responses_only: false overrides the metadata-file default for that backend only.
• Built-in OpenAI registry in src/infrastructure/backends/openai/models/gpt5_family.rs for models that ship with the binary. New entries here should also be reflected in model-metadata.yaml so externally-loaded metadata stays consistent.

After updating any of these sources, restart the router or trigger a hot reload so the new flag takes effect on subsequent requests.

Dispatch behavior¶

The router honors responses_only=true on every public surface that would otherwise hit /v1/chat/completions:

• /v1/chat/completions: requests transparently forward to the upstream /v1/responses endpoint and the response is translated back into a strict-mode chat.completion (or chat.completion.chunk for streaming) envelope.
• /anthropic/v1/messages: the Anthropic-formatted request is converted to the Responses API shape, dispatched to /v1/responses, and the upstream response is translated back into Anthropic Messages JSON (or the Anthropic SSE event sequence for streaming). Tool-call round-trips, web-search emulation, and Unix-socket transports all branch on the flag.

In both cases the dispatch is transparent to the client: the request and response shapes match the surface the client called, so no client-side changes are required to use a responses_only model.

For Chat Completions, the bridge preserves compatible GPT-5 controls including reasoning_effort, flat verbosity, metadata, safety_identifier, prompt_cache_key, prompt_cache_options, and prompt_cache_retention. Invalid values fail explicitly instead of being silently omitted during conversion. On the response side, the /v1/chat/completions to /v1/responses bridge preserves reasoning_content (from reasoning items and summary deltas), refusal (finish_reason: "content_filter"), incomplete_details truncation (finish_reason: "length" or "content_filter"), and upstream failed or cancelled error envelopes across both non-streaming and streaming calls.

Backend-type constraint¶

Only OpenAI and Azure OpenAI backends serve /v1/responses. When a responses_only model is paired with a backend whose type is not OpenAI or Azure OpenAI, the router rejects the request with a 400 invalid_request_error (Anthropic-shaped on /anthropic/v1/messages, OpenAI-shaped on /v1/chat/completions) before any upstream dispatch. The message names both the model and the configured backend type so the misconfiguration is visible from the client log.

The first dispatch per (backend, model) pair logs at info level so operators can confirm Responses-API routing without enabling debug logs. The log line and the responses_bridge_total metric both carry a reason field: responses_only_model for this flag, and tools_with_reasoning for the narrower conditional bridge below.

The narrower flag for function tools with reasoning¶

responses_only is the right flag only when the model has no working Chat Completions route at all. Some models are served on /v1/chat/completions and refuse just one request shape there: function tools combined with a reasoning effort other than none. Setting responses_only on those works, at the cost of sending every request for the model, including plain chat without tools, through the Responses API.

Use chat_completions_tools_require_none_reasoning instead. It bridges only the refused requests and leaves the rest on Chat Completions, and it ships already set on the affected OpenAI models. continuum-router config validate warns when a backend model_configs entry sets responses_only: true on one of them. See Reasoning Effort for the model matrix and the drop-in that clears the flag.

Namespace-Aware Matching¶

The router handles model IDs with namespace prefixes. For example:

• Backend returns: "custom/gpt-5.6-sol", "openai/gpt-5.6-sol", "optimized/gpt-5.6-sol"
• Metadata defined for: "gpt-5.6-sol"
• Result: All variants match and receive the same metadata

This allows different backends to use their own naming conventions while sharing common metadata definitions.

Metadata Priority and Alias Resolution¶

When looking up metadata for a model, the router uses the following priority chain:

1. Exact model ID match
2. Exact alias match
3. Date suffix normalization (automatic, zero-config)
4. Quantization / format suffix normalization (automatic, zero-config; see below)
5. Combined date + format suffix normalization
6. Wildcard pattern alias match
7. Base model name fallback (namespace stripping)

Within each source (backend config, metadata file, built-in), the same priority applies:

1. Backend-specific model_configs (highest priority)

   backends:
     - name: "my-backend"
       model_configs:
         - id: "gpt-5.6-sol"
           aliases: ["gpt-5.6"]
           metadata: {...}  # This takes precedence
   

2. External metadata file (second priority)

   model_metadata_file: "model-metadata.yaml"
   

3. Built-in metadata (for OpenAI and Gemini backends)

Automatic Date Suffix Handling¶

LLM providers frequently release model versions with date suffixes. The router automatically detects and normalizes date suffixes without any configuration:

Supported date patterns:

• -YYYYMMDD (e.g., claude-opus-4-5-20251130)
• -YYYY-MM-DD (e.g., gpt-4o-2024-08-06)
• -YYMM (e.g., o1-mini-2409)
• @YYYYMMDD (e.g., model@20251130)

How it works:

Request: claude-opus-4-5-20251215
         ↓ (date suffix detected)
Lookup:  claude-opus-4-5-20251101  (existing metadata entry)
         ↓ (base names match)
Result:  Uses claude-opus-4-5-20251101 metadata

This means you only need to configure metadata once per model family, and new dated versions automatically inherit the metadata.

Automatic Quantization and Format Suffix Handling¶

Real-world model IDs arriving at /v1/models, routing logic, and backend metadata enrichment frequently combine a canonical base ID with one or more trailing quantization, format, or flavor tokens. The router strips an allowlisted set of such tokens iteratively and retries exact-id, exact-alias, and date-suffix matching after each peel, so you only need to configure metadata for the canonical base ID.

Token Categories¶

The following trailing tokens are detected and stripped (case-insensitive):

Parameter-Count Suffixes are Preserved¶

Tokens that look like parameter counts are never stripped, even when they share a trailing b:

• Kept: -32b, -70b, -8b, -4b, -a3b, -a22b, -0.6b, -1.7b, -e4b
• Stripped: -4bit, -8bit, -16bit (the literal bit suffix marks quantization)

This discrimination ensures that a parameter-count variant like qwen3-32b resolves only to explicit qwen3-32b metadata, never to a generic qwen3 entry via accidental stripping.

Layered Peeling¶

Tokens are stripped one at a time. After each peel, the router re-runs exact-id, exact-alias, and date-suffix match before attempting another peel. This lets alias configurations like gemma-3-4b-it-qat still win even when the request is gemma-3-4b-it-qat-4bit:

Request: gemma-3-4b-it-qat-4bit
         ↓ (peel -4bit)
Try:     gemma-3-4b-it-qat
         ↓ (matches alias of gemma-3-4b-qat)
Result:  Uses gemma-3-4b-qat metadata

A peel that ends on a date suffix is also date-stripped and fed back through the peel loop, so a trailing date does not terminate the chain:

Request: Ministral-3-14B-Instruct-2512
         ↓ (strip the -2512 date)
Try:     ministral-3-14b-instruct
         ↓ (peel -instruct)
Result:  Uses ministral-3-14b metadata

Priority Note¶

Stripping runs after exact-id and exact-alias match. A canonical base ID that happens to end in an allowlisted token (for example gemma-3-12b-qat) wins before the peel phase runs, so existing configurations remain stable.

Suffix-Order Ambiguity¶

Both -qat-4bit and -4bit-qat orderings appear in real-world model IDs, and peeling removes one token at a time from the right:

• Peel sequence: gemma-3-12b-qat-4bit peels as gemma-3-12b-qat-4bit -> gemma-3-12b-qat -> gemma-3-12b, whereas gemma-3-12b-4bit-qat peels as gemma-3-12b-4bit-qat -> gemma-3-12b-4bit -> gemma-3-12b.
• Shared QAT metadata: If both suffix orderings should resolve to the same QAT metadata, configure the canonical QAT base ID (gemma-3-12b-qat) with the QAT metadata and keep a separate entry for the non-QAT ID (gemma-3-12b); the deepest match wins at each peel depth.
• Distinct variant metadata: When QAT and non-QAT variants require distinct tier or capability metadata, define explicit aliases enumerating both orderings rather than relying on peel order alone.

Length Bounds¶

The layered peel phase caps input length at 256 characters and iteration count at 8 peels as defense-in-depth against pathological inputs. Matching still runs (the exact-id and exact-alias phases remain in effect), but the peel phase short-circuits instead of walking a long allowlist-token chain. Request handlers enforce the same 256-character cap on the model field for every chat / completion / embedding endpoint, so normal traffic never hits the internal cap.

Case Insensitivity¶

Stripping is case-insensitive, so Qwen3.5-4B-4bit, QWEN3.5-4B-4BIT, and qwen3.5-4b-4bit all resolve to the same qwen3.5-4b metadata entry. Exact-id and exact-alias match phases (1 and 2) remain case-sensitive, so HuggingFace-style aliases like BAAI/bge-m3 keep their original behavior.

Wildcard Pattern Matching¶

Aliases support glob-style wildcard patterns using the * character:

• Prefix matching: claude-* matches claude-opus, claude-sonnet, etc.
• Suffix matching: *-preview matches gemini-3.1-pro-preview, o1-preview, etc.
• Infix matching: gpt-*-turbo matches gpt-4-turbo, gpt-3.5-turbo, etc.

Example configuration with wildcard patterns:

models:
  - id: "claude-opus-4-5-20251101"
    aliases:
      - "claude-opus-4-5"     # Exact match for base name
      - "claude-opus-*"       # Wildcard for any claude-opus variant
    metadata:
      display_name: "Claude Opus 4.5"
      # Automatically matches: claude-opus-4-5-20251130, claude-opus-test, etc.

  - id: "gpt-5.6-sol"
    aliases:
      - "gpt-5.6"             # Exact match for the catalog alias
      - "gpt-5.6-*"           # Wildcard for any GPT-5.6 variant
    metadata:
      display_name: "GPT-5.6 Sol"

Priority note: Exact aliases are always matched before wildcard patterns. When both could match, the exact alias wins.

Alias Dispatch¶

Metadata resolution determines pricing, capabilities, and /v1/models entries, whereas backend selection matches the requested model name literally against each backend's models: list and the live catalog. When a client requests a metadata alias that no backend serves under that literal name, the router rewrites the request's model field to the canonical ID before selecting a backend if all four conditions hold:

1. Exact alias match: The requested name (or its models/-prefixed form) is declared as an exact aliases entry for a different canonical ID in an enabled backend's model_configs or in model-metadata.yaml. Fuzzy metadata phases (date suffix, quantization peel, HuggingFace prefix, and wildcard aliases) do not trigger dispatch rewrites.
2. Discovery complete: Every enabled backend whose models: list names the requested ID, by an exact entry or a glob entry that matches it (an entry that matches every name, such as *, is excluded), has populated at least one entry in the live catalog (backends still running initial discovery are not treated as evidence of absence).
3. Alias not served literally: No backend serves the requested name in the live catalog (discovered via GET /v1/models or statically configured when discovery is disabled).
4. Canonical ID available: At least one user-routable backend serves the canonical ID in the live catalog.

When all four conditions are met, the router dispatches using the canonical ID:

• Canonical rewrite: Replaces model with the canonical ID and logs Dispatching a model alias as the canonical id its backend serves at info level (for example, rewriting gemini-3.1-pro to gemini-3.1-pro-preview when Google serves only the -preview ID). Because the upstream backend responds with the model ID it served, the response model field contains the canonical ID.
• Literal match precedence: If any backend serves the requested name literally (such as a local engine advertising unsloth/Qwen3.6-35B-A3B-GGUF), the name is never rewritten; if no backend serves either name, the original name is kept and fails with model-not-found.

Note: A glob entry such as openai/* in models: on a backend with no catalog entries keeps every name it matches, so alias dispatch is off for those names. A backend that uses a static models: list (discovery disabled) never publishes glob entries in its catalog, so a backend whose list holds only such entries keeps those names permanently. An entry that matches every name (* or models/*) does not count here: it never holds a name back, including while its backend's discovery is still running. List exact model IDs, or let discovery populate the catalog, to keep alias dispatch active for names a narrower glob matches.

Warning: A match-all entry (* or models/*) does not keep requests on its backend. A request for a metadata alias is rewritten to its canonical ID and sent to a backend that serves that ID, which may be a remote provider. A plain model name that is not an alias is not rewritten. If requests must stay on the backend, list the names exactly or use a narrower glob. continuum-router config validate warns about a match-all entry when another backend is enabled.

Interaction with other features:

• Access control and parameter policy: Per-key allowed_models lists and request_params.models scopes are evaluated against the original model name requested by the client, so keys permitted to use the alias continue to work after the rewrite.
• Model lists: Each aliases entry in an enabled backend's model_configs is published as its own entry on GET /v1/models, /v1/models/extended, /v1/models/{model}, and /anthropic/v1/models while the four conditions above hold, with the canonical entry's backends and metadata. See Backend aliases in model lists. Aliases from model-metadata.yaml and the model_aliases section are not listed.
• Fallback chains: fallback.fallback_chains is looked up using the rewritten canonical ID. Key fallback chains on the canonical ID rather than the alias; the router logs a warning at request time if it detects a fallback chain keyed on a rewritten alias.
• Pinned routing: Hub-attributed requests, requests under a Hub exact-model budget guard, and AppProxy ingress-pinned requests already carry an explicit model decision and are never rewritten.
• Covered endpoints: /v1/chat/completions, /anthropic/v1/messages, /anthropic/v1/messages/count_tokens, /v1/responses, /v1/responses/compact, /v1/embeddings (including native Gemini multimodal embeddings), /v1/images/generations, /v1/images/edits, /v1/images/variations, and the Realtime WebSocket handshake. ACP remains literal-only (see ACP Usage and ACP Architecture).
• Error reporting and image endpoints: Any 404 or 403 returned after a rewrite reports the model name the client originally requested. On /v1/images/edits and /v1/images/variations, the fixed allowlist of supported image model names is checked before alias rewriting, so an alias is accepted there only if the alias itself is on that list.

Per-Backend Upstream Model IDs¶

A backend can serve a model under a different ID from the one clients use. Declare upstream_id on that backend's model_configs entry: the public id remains the routing identity, and only the request sent to that backend carries the upstream spelling. This lets one public name pool backends that disagree about the model's name, such as an in-house vLLM deployment with a quantized checkpoint ID and an external OpenAI-compatible service.

backends:
  - name: "in-house-vllm"
    type: vllm
    url: "http://localhost:8000"
    model_configs:
      - id: "gemma4"
        upstream_id: "gemma-4-26b-a4b-it-fp8"
  - name: "external"
    type: openai
    url: "https://api.example.com"
    api_key: "${EXTERNAL_API_KEY}"
    models: ["gemma4"]

• Routing identity: Selection, per-key allowed_models and request_params scopes, model lists, fallback-chain lookup, and the usage ledger use the public id. A backend without upstream_id keeps receiving the public ID.
• Request rewrite: Just before each attempt is serialized, the router replaces the body's model with upstream_id for the backend that will receive it, and the models/{id} path segment on native Gemini calls. Retry and fallback hops rewrite again for the backend they land on.
• Response restore: A response whose top-level model carries the upstream ID is renamed back to the public ID. Buffered JSON bodies are restored at the dispatch seam, and OpenAI-wire SSE chunks by the streaming relay, so the Anthropic and Responses ingresses see the public ID as well.
• Catalog: A backend's live model list contributes the public ID, so the pool publishes one entry for the name. The upstream ID is not listed and is not routable: a request naming it fails with model-not-found unless some backend lists it literally.
• Validation: config validate rejects a blank upstream_id and a value that collides with another model_configs[].id on the same backend. Different backends may map the same public ID to different upstream IDs, which is the pooling case.

Note: upstream_id is not an alias. Aliases are alternative names a client may request; an upstream ID is only ever sent to its own backend, never accepted as a request name unless a backend publishes it through models or model_configs.

Using Aliases for Model Variants¶

Aliases are particularly useful for:

• Different quantizations: qwen3-32b-i1, qwen3-23b-i4 → all use qwen3 metadata
• Version variations: gpt-4-0125-preview, gpt-4-turbo → share gpt-4 metadata
• Deployment variations: llama-3-70b-instruct, llama-3-70b-chat → same base model
• Dated versions: claude-3-5-sonnet-20241022, claude-3-5-sonnet-20241201 → share metadata (automatic with date suffix handling)

Example configuration with aliases:

model_configs:
  - id: "qwen3"
    aliases:
      - "qwen3-32b-i1"     # 32B with 1-bit quantization
      - "qwen3-23b-i4"     # 23B with 4-bit quantization
      - "qwen3-16b-q8"     # 16B with 8-bit quantization
      - "qwen3-*"          # Wildcard for any other qwen3 variant
    metadata:
      display_name: "Qwen 3"
      summary: "Alibaba's Qwen model family"
      # ... rest of metadata

Aliases vs. suffix normalization: when to use which¶

Two coverage layers resolve non-canonical model ids to their owning metadata entry: explicit YAML aliases, and the layered suffix-peel allowlist in src/models/pattern_matching.rs. They are complementary, not redundant. This section explains how to choose between them when adding a new entry.

Matching phase order¶

The pipeline runs in this order, and a successful match in an earlier phase short-circuits the later ones:

1. Exact model id (case-sensitive).
2. Exact alias (case-sensitive).
3. Date-suffix normalization (-YYYYMMDD, -YYYY-MM-DD, -YYMM, @YYYYMMDD).
4. Layered quantization / format / flavor peel (case-insensitive; after each peel, exact-id + exact-alias + date-suffix phases re-run; combined date + format handled in the same loop).
5. HuggingFace repo-prefix stripping (vendor/repo -> repo) with re-entry into phases 1-4 on the stripped residual. Single-hop re-entry; phase 5 does not recurse.
6. Wildcard alias (glob-style * patterns).

A retained explicit alias runs in phase 2, strictly before the peel (phase 4) and before the prefix-strip layer (phase 5). When a retained alias and a peel-or-strip path would resolve to different metadata, the alias wins deterministically. Aliases are therefore a stronger intent signal than peel-or-strip coverage, not a weaker one.

The three alias classes¶

Every alias in model-metadata.yaml falls into one of three classes.

peel-coverable¶

Normalization reaches the same owner id without the alias, and the target metadata is the correct one. These are deletion candidates. Example: qwen3.6-35b-a3b-instruct as an alias of qwen3.6-35b-a3b. Phase 4 peels the FLAVOR token -instruct and lands on the base id directly, so the explicit alias adds no coverage. When deleting an alias in this class, add a regression assert in tests/format_suffix_normalization_test.rs::real_metadata_removed_aliases_still_resolve so the peel coverage that replaces it stays locked in.

vendor-prefix¶

The alias carries a vendor or repo prefix that suffix peel cannot strip, because peel only removes right-side tokens from a closed allowlist. Phase 5 (HuggingFace prefix stripping with re-entry into phases 1-4, where phase 4 is case-insensitive) resolves the mixed-case HF form even without the alias. Example: Qwen/Qwen3.6-35B-A3B as an alias of qwen3.6-35b-a3b. Phase 2 wins on the explicit alias, but phase 5 would also reach the base id via Qwen/ -> residual Qwen3.6-35B-A3B -> phase 4 case-insensitive match. These aliases are peel-coverable-adjacent: phase 5 makes them redundant for resolution, but they remain a deterministic intent signal. Keep them with a YAML comment noting the covering phase.

intentional-override¶

The alias deliberately routes a differently-weighted model under another entry's metadata, as an operator decision. Keep. Example: smoothie-qwen3-32b-i1 as an alias of smoothie-qwen3. The smoothie-qwen3-32b-i1 fine-tune has its own weights; the operator has chosen to surface it under the umbrella smoothie-qwen3 metadata rather than give it a dedicated entry. Peel must not infer this equivalence on its own. When an alias sits in this class, the YAML comment on the line must note that the underlying weights differ from the owner id, distinguishing an intentional override from a mechanical normalization gap.

Guidance for adding a new alias¶

Before adding a line to model-metadata.yaml, ask whether peel already covers it.

• If the new id is a canonical base with a trailing quantization, format, or flavor token already in the allowlist, and the weights are equivalent to the base metadata, do not add the alias. The peel handles it, and adding the alias would be dead code.
• If the new id shares weights with the base but ends in a token class that peel does not handle (for example, a novel fine-tune label like -abliterated or a new quantization format like -nf4), prefer extending the peel allowlist in src/models/pattern_matching.rs. This is a code change with test coverage, and it covers the entire token class in one move.
• If the new id has a vendor prefix, a repo namespace that normalization would not case-match, a parameter-count token blocking the peel chain (-Nb, -aNb, -eNb), or intentionally-different weights, add the alias with a YAML comment that states the reason. If weights differ from the owner id, say so in the comment.

Surface distinction: code-gated vs. YAML-gated¶

The peel allowlist is the strategic layer. Aliases are the tactical override and emergency channel.

Token categories already on the peel allowlist¶

The allowlist in src/models/pattern_matching.rs currently covers:

• BIT_WIDTH: -2bit, -3bit, -4bit, -5bit, -6bit, -8bit, -16bit
• GGUF_QUANT: -Q4_K_M, -Q4_K_S, -Q5_K_M, -Q6_K, -Q8_0, -Q2_K, -IQ2_XS, -IQ3_XXS, -IQ4_XS, -F16, -F32, -BF16
• FP_FORMAT: -FP4, -FP8, -FP16, -FP32, -NVFP4, -MXFP4, -MXFP8
• INT_FORMAT: -INT2, -INT4, -INT8
• WEIGHT_ACTIVATION: -W<bits>A<bits> (-W4A16, -W8A8, -W4A8), also accepted with the quantized. prefix (-quantized.w4a16)
• SCALING_GRANULARITY: -block, -dynamic, -static, peeled only after an FP / INT / weight-activation token
• LIBRARY: -AWQ, -GPTQ, -BNB, -HQQ, -EXL2, -EXL3, -MLX
• IMATRIX: -i1 through -i8, -q2 through -q8
• UNSLOTH: -UD-Q<digit>_<KIND>, -UD-IQ<digit>_<KIND>, -unsloth
• CONTAINER: -GGUF, -GGML, -SAFETENSORS, -ONNX
• FLAVOR: -it, -instruct, -chat, -base, -thinking, -qat
• HARDWARE: -rngd, -warboy, -atom, -atommax, -rebel

Tokens that mark a behavior-changing derivative are deliberately absent, because the peeled base would be a different logical model: -abliterated, -heretic, -uncensored, -distilled, -REAP-<params>, -DFlash / -DSpark, -speculator.eagle3, -MTP, -assistant.

Parameter-count suffixes (-Nb, -aNb, -eNb, -0.6b, -1.7b) are never peeled. They are part of canonical model identity and terminate the peel chain. This is why qwen3-32b-i1 must be kept as an explicit alias of qwen3: phase 4 strips -i1 and then halts at -32b, so without the alias the chain exhausts before reaching the base id.

HuggingFace repo-prefix stripping (phase 5)¶

Phase 5 normalizes HuggingFace-style vendor/repo prefixes off the left side of a model id, complementing the right-side suffix peel. It resolves the common HF-GGUF class where a user submits an id like unsloth/Qwen3.6-35B-A3B-GGUF and expects it to route to the canonical qwen3.6-35b-a3b metadata, without an explicit alias for every vendor x base x quant combination.

How phase 5 runs¶

1. The input is inspected for a / separator. No /, no-op.
2. Total segments (count of / plus one) must be at most MAX_PREFIX_SEGMENTS (3). org/team/repo is permitted; a/b/c/d/model is rejected outright.
3. All segments must be non-empty and free of ASCII whitespace. Malformed inputs like /repo, vendor/, vendor//repo, or vendor /repo are rejected.
4. On success, the residual is the substring after the last /. This residual is fed back into phases 1-4 with the re-entry gate closed. Phase 5 does not recurse: the inner call cannot trigger phase 5 again, so the recursion depth is exactly 1 by construction.

Composition with suffix peel¶

The re-entry runs through phase 4, so prefix stripping composes with suffix peel in a single lookup. unsloth/Qwen3.6-35B-A3B-GGUF strips to Qwen3.6-35B-A3B-GGUF, phase 4 peels -GGUF, case-insensitively matches qwen3.6-35b-a3b. This is the motivating case for the phase and covers HuggingFace GGUF forks without requiring hand-enumerated aliases.

Registered-alias precedence¶

Operators who explicitly register a vendor/repo form as a YAML alias keep deterministic control. Phase 2 runs before phase 5, so the exact alias wins before the stripping layer ever considers the input. Use this when the prefixed form must route to a different metadata entry than the canonical base id would.

Out of scope¶

• Hyphen-delimited vendor prefixes (e.g., smoothie-qwen/smoothie-qwen3-32b-i1). Different semantic class, different detection difficulty, and often represents different weights where silent base-metadata routing is the wrong call.
• Automatic vendor discovery from the HuggingFace API. The layer is purely syntactic.
• Extending the suffix peel allowlist. Orthogonal change; follow the peel-extension path for novel token classes.

Security bounds¶

Parallels the suffix peel:

Phase 5 is constant-time on adversarial input: after the segment-count, emptiness, whitespace, and length guards, the work reduces to a single slice lookup plus one additional pass through phases 1-4.

Audit procedure¶

To re-audit the YAML, run:

cargo test --test alias_audit_helper -- --ignored --nocapture audit_metadata_aliases

The helper prints every alias with its classification (REDUNDANT, LOAD-BEARING-DRIFT, LOAD-BEARING-LOSS, or WILDCARD) and the post-removal resolution target.

API Response¶

The /v1/models endpoint returns enriched model information:

{
  "object": "list",
  "data": [
    {
      "id": "gpt-5.6-sol",
      "object": "model",
      "created": 1234567890,
      "owned_by": "openai",
      "backends": ["openai-proxy"],
      "metadata": {
        "display_name": "GPT-5.6 Sol",
        "summary": "Frontier GPT-5.6 model for complex professional work",
        "capabilities": ["chat", "vision", "code", "reasoning", "tool"],
        "knowledge_cutoff": "2026-02",
        "pricing": {
          "input_tokens": 5.0,
          "output_tokens": 30.0
        },
        "limits": {
          "context_window": 1050000,
          "max_output": 128000
        }
      }
    }
  ]
}

Request Parameter Policy¶

Use the optional request_params section to define typed defaults, operator-selected overrides, and limits for the request-parameter policy. The Router validates and hot-reloads this configuration snapshot, then applies it once at Chat Completions, Completions, Responses, and Anthropic Messages ingress before cache identity or provider routing is chosen.

request_params:
  defaults:
    max_tokens: 1024
  overrides:
    temperature: 0.0
  limits:
    max_tokens:
      min: 1
      max: 4096
  models:
    gpt-5:
      defaults:
        top_p: 0.9
      limits:
        max_tokens:
          max: 2048

Maintained examples use block-style YAML. Equivalent flow mappings such as { max_tokens: 1024 } are also accepted, and TOML configuration has the same semantics.

Evaluation Order¶

Protocol integrations apply the three modes in a fixed order:

1. defaults fills a missing or JSON/YAML-null field and preserves a value supplied by the client.
2. overrides replaces the resulting value, whether it came from the client or a default.
3. limits clamps an existing value to the inclusive min and max bounds. A limit does not create a missing field.

The top-level defaults, overrides, and limits blocks define the global policy. Each models.<name> entry can define the same three blocks for a specific model:

• Model-specific fields override global fields independently within each mode; omitted fields inherit the global value or range.
• Model names must match the resolved model ID after local alias resolution (backend scopes, regular expressions, and globs are not supported).
• The selected scope is locked before control-plane substitution, arbitrage, backend selection, retry, or fallback, and stays fixed across retries and fallback hops.
• When control_plane.policy delivers Hub tier limits, those limits are intersected with the local limits bounds (neither side can widen the other, and an empty intersection rejects the request).

Supported Parameters and Validation¶

Only the following canonical parameter names are accepted:

Unknown modes or parameter names, wrong scalar types, non-finite floats, out-of-range values, and min > max are rejected with a path-qualified error. models accepts at most 64 exact scopes; each model key must be non-empty after trimming, at most 256 bytes, and free of control characters. The same parameter may appear in more than one mode because the evaluation order is deterministic. A null policy value is treated as omitted; it is not a request to force JSON null.

Runtime Provider Compatibility¶

Compatibility checks apply only to fields that the policy configured and actually changed or constrained. Client fields that the policy did not govern retain their existing provider-specific passthrough behavior. If an eligible primary or configured fallback route cannot preserve a policy-effective field, the Router rejects the request before cache lookup or upstream I/O with unsupported_request_parameter instead of silently dropping the enforced value.

On the native Anthropic Messages ingress, a native Anthropic backend receives the client's anthropic-version, or the backend's anthropic_version (default 2023-06-01) when the client sent none. The Chat Completions and Responses conversions always send the backend's anthropic_version (see "Anthropic API Version" in the backends guide). Under 2026-10-21 or later, Anthropic rejects temperature, top_p, and top_k on every model, so those fields count as unsupported on such a backend:

• Policy writes: Defaults and overrides for these fields are rejected before upstream I/O. The error names the anthropic-version cause.
• Limits on absent fields: A limit constrains nothing when the request lacks the field. Native Messages also treats client sampling as absent when every eligible backend drops it. Mixed provider candidates retain the conservative capability check.
• Fallback policy: A hop is refused if it would drop a policy-written sampling value or forward a client value that was hidden from a sampling limit on the primary.
• Client values: Native Messages, Chat Completions, and Responses remove client sampling on Anthropic hops that forbid it. Native Messages preserves the original values for later hops to providers that support them.
• Configuration check: continuum-router config validate warns at backends[N].anthropic_version when a 2026-10-21 backend serves a model that still honors sampling under 2023-06-01 (Claude Opus and Sonnet below 4.7, Haiku 4.5) and this section sets a sampling field for it.

Streaming fallback and caching:

• Pre-stream fallback: Re-enters typed per-backend dispatch, so hops onto native Anthropic, Bedrock, or Gemini backends run through that provider's native pipeline with a single wire-format conversion.
• Mid-stream fallback: Once SSE output has begun, recovery uses only the OpenAI-compatible wire format. Chain entries that resolve to native-protocol or Unix-socket backends are skipped in favor of the next entry rather than receiving a partially translated request. Every hop is rebuilt from the canonical effective request so multi-hop fallbacks never accumulate conversions or drop restored fields.
• Response cache isolation: Cache lookup and storage run only when routing resolves to a single policy-compatible backend with no active fallback chain, and the backend identity is included in the cache namespace. Requests with multiple eligible backends or an enabled fallback chain bypass exact, prefix, and streaming caches.

Observability and Enforcement¶

When policy changes at least one request field and the selected provider path can preserve the effective request, the Router adds the informational x-continuum-request-params response header. Its comma-separated tokens contain only a bounded action and canonical parameter name, such as default:max_tokens or limit:temperature; request values, model names, keys, and tier identifiers are never included.

The policy is mandatory operator configuration. Clients cannot disable or bypass it with a request header, query parameter, or request-body option. The informational response header does not provide an opt-out and is not an authorization credential.

The header is present on successful responses, compatible cache hits, and upstream error responses produced after an effective mutation. It is omitted when policy is a no-op and when policy or provider compatibility rejects the transaction. A rejected transaction is rolled back and never receives a misleading applied header. Logs and Prometheus metrics follow the same bounded applied/rejected outcomes.

Hot Reload, Config Sync Precedence, and Privacy¶

request_params takes effect immediately on hot reload without restarting the server. In control-plane builds with control_plane.config_sync.enabled: true, Continuum Hub can also synchronize routing and fallback settings alongside request_params:

• Hub-managed fields:
  • request_params policy defaults, overrides, limits, and model scopes.
  • Public kv_routing leaves: selection_strategy, prefix_routing.{enabled,max_prefix_length,load_factor_epsilon,virtual_nodes,anthropic_cache_control_injection}, and kv_cache_index.{enabled,max_entries,entry_ttl_seconds,scoring.*,storage_offloading.*,event_sources.<backend>.{endpoint,reconnect_interval_ms}} for transport: sse sources.
  • Model fallback switches, fallback_chains, fallback_policy, model_settings, and empty_reply_retry.
• Deployment-local fields (never observed or overwritten by Hub):
  • prefix_routing.salt_echo, kv_cache_index.backend, and kv_cache_index.event_sources entries with transport: event_bus. Local values set in config.yaml or via the Admin API survive Hub config-sync applies, rebases, and restarts without pinning the managed leaves.
  • fallback.quality_check, which always remains local and is excluded from Fleet observation and digests.
• Observed: fallback.fallback_policy.trigger_conditions.context_window_exceeded is observed and moves digests. It is not deployment-local, so a Hub that registers the observed path may publish it. This Router release requires such a Hub: an older Hub rejects the whole heartbeat over the unknown path, and staying silent would leave the Hub's desired and observed digests apart as permanent drift.
• Not observed yet: fallback.empty_reply_retry is applied but not observed, and does not move digests. It is not deployment-local, and a hub may publish it later.
• Local pinning precedence:
  • Set request_params_immutable, kv_routing_immutable, or fallback_immutable under control_plane.config_sync to pin an entire section locally.
  • A local fallback block in config.yaml pins its switches and fallback_policy as a unit, and each local entry in fallback_chains or model_settings pins that complete collection.
• Atomic reload and in-flight isolation:
  • A validated revision atomically replaces the active configuration snapshot and reconciles the process-lifetime fallback service before the next request (including enable/disable toggles, exact clears, and dial-bound updates).
  • A rejected revision leaves the last-known-good snapshot active.
  • Each request captures a single immutable config snapshot and authenticated Hub tier policy before mutation, so in-flight requests, retries, and fallback hops never switch policies mid-request.
• Privacy: Applied-policy telemetry records only bounded parameter names, actions, composition sources, and outcomes, never request values. Admin config and MCP views may display the operator-authored policy, but they never serialize prompt content or request payloads and always redact secret-typed configuration fields.

Responses Tool Policy¶

A hosted tool runs on the backend and is billed to your upstream key, and on a self-hosted backend it may reach into your network. The client picks these tools per request, so responses_tools decides which types POST /v1/responses lets through. It applies to every backend type, before backend selection and any cache lookup.

responses_tools:
  unmodeled_types: deny        # allow | deny
  modeled_hosted_types: allow  # allow | deny
  allow: [image_generation]    # exact wire types, overrides the two defaults
  deny: [web_search]           # wire types in any ASCII case, overrides the two defaults
  deny_resource_fields: [vector_store_ids, container_ids]  # default

Each tool is judged in this order, using its type string:

1. function, custom, local_shell, namespace, and tool_search with execution: "client" run on the client and are always allowed. A namespace is only a container: each member is judged by these same rules, at any depth.
2. A type in deny is refused.
3. A type in allow is forwarded.
4. web_search, file_search, code_interpreter, computer_use, and programmatic_tool_calling follow modeled_hosted_types. Every other type, such as mcp, image_generation, or tool_search with another execution, follows unmodeled_types.

Listing tool_search in deny refuses only the backend-executed forms: tool_search with execution: "client" stays allowed under rule 1. deny ignores ASCII case, so deny: [mcp] also refuses MCP. allow and the lists in rule 4 match exactly, so a case variant can be refused but never admitted. The same walk judges every nested list a forwarded tool declares: a tools list on a tool_search, a tools list on a programmatic_tool_calling, and the tools a raw tool_choice carries, such as allowed_tools. A tools value that is not an array is refused as tools[unknown].

A refused tool gets 400 unsupported_request_parameter naming it, for example tools[mcp] or tools[mcp] member of namespace 'ns'. The router does not make an upstream request, and it does not remove the tool and forward the rest. A nested namespace it cannot read is refused as tools[unknown].

A refusal writes a refused prompt-log record when prompt logging captures the request, and its log line carries the caller's key ID, never the key.

An entry must be printable ASCII. An entry carrying whitespace, a control character, or an invisible look-alike such as a zero-width space or a soft hyphen is a load error, so a name that could never match is rejected instead of silently doing nothing. The router also reports a load error for an empty entry, an entry in both lists, and an entry naming an always-allowed type, the last two compared ignoring ASCII case. An edit applies to the next request without a restart.

Client-Supplied Resource IDs¶

On a router that shares one upstream key among tenants, a client holding a resource ID can reach that resource under your key. deny_resource_fields refuses these references with 400 unsupported_request_parameter naming the field, for example tools[file_search].vector_store_ids, and makes no upstream request.

The two default entries also apply to namespace members and to the tools list of a tool_search, of a programmatic_tool_calling, and of a raw tool_choice. A single-tenant router that uses native file_search or a named container must set deny_resource_fields: []. An unknown entry is a load error.

input_item_resource_ids walks every input item, including its content parts and the extra fields a typed item carries, and refuses the first value that names a resource, naming the location as input[code_interpreter_call].container_id or input[message].file_id. A key set to null, an empty string, or an empty array names nothing and passes. A file_id the router resolves to a local file is cleared before the check, so a request that names a local file is accepted either way.

A shared-key router should consider listing the other five. They are off by default because they are long-standing and because a client replaying the model's own output resends those IDs, so refusing previous_response_id or replayed input items breaks ordinary multi-turn clients. These entries do not change a conversion strategy, which handles or ignores those fields on its own.

A request the pass-through strategy would refuse is neither served from nor stored to the response cache, so a reload that lists a new entry takes effect immediately instead of replaying an answer cached under the previous policy. An entry cached before the change ages out with the cache TTL.

The router does not check whether a caller owns an upstream resource, and this list does not cover request_extensions.body or GET /v1/responses/{id} and its cancel route.

What the Policy Does Not Cover¶

The policy does not cover Chat Completions or Anthropic Messages, tools attached by a stored prompt, tools you set in request_extensions.body, or mcp.server_url values. A client-executed type that a future client adopts gets 400 until you add it to allow or a router release adds it to the always-allowed list.

Request Presets¶

A request preset is an operator-defined bundle of a model default and sampling values that a client selects by slug. Only POST /v1/chat/completions reads the field today; the other endpoints do not support it yet.

Configuration¶

request_presets:
  support-bot:
    model: gpt-5-mini        # optional: used only when the request carries no model
    params:                  # optional: the same seven canonical fields as request_params
      temperature: 0.2
      max_tokens: 1024

Validation rules:

• A slug is 1 to 64 characters of [A-Za-z0-9._-], and a configuration declares at most 256 presets.
• model must be non-empty after trimming, at most 256 bytes, and free of control characters.
• Every preset must set model, params, or both.
• params accepts only temperature, top_p, max_tokens, presence_penalty, frequency_penalty, top_k, and min_p, with the same value bounds as request_params.
• Unknown preset or parameter keys are rejected at load and at hot reload.

Request Resolution¶

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"preset": "support-bot", "messages": [{"role": "user", "content": "Hello"}]}'

• Priority: a field set explicitly on the request wins over the preset, and the preset wins over request_params.defaults. request_params.overrides and limits still apply to preset-supplied values, because the ordinary policy pass runs after the preset.
• Model: the preset model is used only when the request has no usable model; a valid request model always wins. A per-request models list counts as a request model: its first entry is the primary when model is absent, so the preset model is not used.
• Admission: when the request has no model, the per_model rate limit and the Continuum Hub model gate judge the model the request will run under (the first models entry, otherwise the preset model), so naming a preset cannot skip a per-model limit.
• Validation order: the client's own max_tokens is type- and range-checked before the preset's sampling values are filled, so a mistyped budget answers the ordinary 400 invalid_type naming that field.
• Dispatch: the router removes the preset field before dispatch, so no backend ever receives it.

Errors¶

Hot Reload and Hub Sync¶

request_presets hot-reloads immediately: a new request uses the new presets without a restart. It is a separate top-level section from request_params, which the Hub config sync replaces as a whole, so a Hub apply never drops local presets.

The feature corresponds to OpenRouter Presets, but the model: "@preset/<slug>" spelling is not supported; name the preset in the preset field instead.

Hot Reload¶

When the hot-reload feature and file watcher are active, a valid file revision is published as a new immutable configuration snapshot. Invalid revisions leave the previous snapshot active. Not every parsed field has a live runtime consumer, so use the conservative matrix below.

Applied without restart¶

• Backend additions/removals/edits (the pool is reconciled; new backends receive an immediate health check)
• Health-check intervals and thresholds
• Rate-limit and circuit-breaker policies when their services are active
• Global-prompt, request-parameter, request-preset, and streaming policy read by new requests
• The responses_tools hosted tool policy read by new requests
• All fallback switches, chains, policy, per-model settings, and dial bounds; the process-lifetime service moves between dormant and live states
• Smart-routing policy snapshots
• selection_strategy (swapped onto the live pool atomically; membership, stats, and in-flight accounting are preserved)
• Retry policy (new requests capture the updated complete policy; in-flight loops retain their entry snapshot)
• Per-request, streaming, image, model-override, and health-check timeout budgets
• All prefix_routing.* fields: enabled and max_prefix_length for request-time prefix extraction; load_factor_epsilon and virtual_nodes swapped onto the live pool (a virtual_nodes change also rebuilds the hash ring); anthropic_cache_control_injection and salt_echo read per-request
• routing.engine_load.max_staleness_intervals and routing.engine_load.admission.* (applied by the engine-stats configuration watcher on each published reload)

Restart required or currently not live¶

• server.bind_address, server.socket_mode, server.workers, and server.shutdown_drain_timeout
• server.connection_pool_size and CORS router layers
• Logging subscriber format/filter and tracing middleware construction
• Toggling optional services that were absent at startup, including fallback.enabled
• Response-cache backend/object construction
• timeouts.connection and timeouts.request.streaming.total (the shared HTTP-client connection/overall ceilings; request budgets still reload live)
• routing.engine_load scoring fields (enabled, engine_load_weight, balance_abs_threshold, balance_rel_threshold): the KV overlap scorer captures them at construction

After editing any restart-required field, validate and restart:

continuum-router config validate config.yaml

Check watcher availability and the router's capability summary through the authenticated Admin API:

curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8080/admin/config/hot-reload-status

Treat the active subsystem state, rather than the newly parsed file alone, as authoritative when a restart-required value changes.

Distributed Tracing¶

Continuum Router supports distributed tracing for request correlation across backend services. This feature helps with debugging and monitoring requests as they flow through multiple services.

Everything in this section is about header propagation: which identifiers the router reads from a request and which it forwards to backends. It is always active and needs no special build. To additionally export the router's own spans to an OpenTelemetry collector, configure the tracing.otlp subsection documented in Distributed Trace Export. That subsection is off by default and requires a binary built with the otel Cargo feature; leaving it out changes nothing about the behavior described here.

Configuration¶

tracing:
  enabled: true                         # Enable/disable distributed tracing (default: true)
  w3c_trace_context: true               # Support W3C Trace Context header (default: true)
  headers:
    trace_id: "X-Trace-ID"              # Header name for 32-hex trace ID (default)
    request_id: "X-Request-ID"          # Header name for client request ID (default)
    correlation_id: "X-Correlation-ID"  # Header name for correlation ID (default)
    client_ref: ""                      # Optional header name for client reference ID, e.g. "X-Client-Ref" (default: "")

How It Works¶

1. Trace ID Resolution (trace_id): When a request arrives, the router resolves a W3C-valid 32-character lowercase hexadecimal, non-zero trace_id ([0-9a-f]{32}) in the following priority order:

   • Valid inbound W3C traceparent header (when w3c_trace_context is enabled)
   • Valid 32-hex non-zero value from the configured trace_id header (X-Trace-ID)
   • Newly generated 128-bit random 32-hex trace ID (Uuid::new_v4().simple())

2. Client Request ID (X-Request-ID) and Continuum Request ID (x-continuum-request-id):

   • Inbound X-Request-ID values matching ^[A-Za-z0-9._:/-]{1,128}$ are accepted and logged as x_request_id without overwriting the 32-hex trace_id. Invalid values are rejected.
   • Every response handled inside the request-ledger middleware scope (streaming 200, non-streaming 200, and 4xx/5xx responses) carries x-continuum-request-id: req_<32 hex>, matching the request ledger record and the router.request_id span attribute. Responses that API-key authentication, Hub policy enforcement, or the identity rate limiter refuse before the ledger scope (401, 403, 429) carry the same headers, so a failed call can be quoted by id; the id then matches the request_id of the event=request_rejected log line, since no usage record exists. Refusals by layers that run ahead of the API routes (the global and per-client limiters 429, the concurrency ceiling 503, and the router hop guard 400 or 508) carry the same headers too.
   • Response x-request-id echoes the client's validated X-Request-ID when present, or falls back to x-continuum-request-id.
   • Responses on the Anthropic Messages API routes (/anthropic/...) also carry request-id, the header the Anthropic API returns, with the same value as x-continuum-request-id. It holds the router id, never the echoed client X-Request-ID, so an Anthropic client that records request-id can match the request ledger record. When a native Anthropic backend returned its own request-id, the router relays that value unchanged instead, and x-continuum-request-id still carries the router id.

3. Optional Client Reference Header (client_ref):

   • When tracing.headers.client_ref is set (for example "X-Client-Ref"), inbound values matching ^[A-Za-z0-9._:-]{1,128}$ are logged as client_ref, exported on the router.request span as router.client_ref, and forwarded to the backend under the configured header name.
   • Invalid values are dropped without failing the request and increment client_ref_rejected_total{reason} (empty, too_long, control_char, or invalid_char).

4. Header Propagation: Outbound requests to backends carry:

   • X-Request-ID, X-Trace-ID, and X-Correlation-ID: Set to the 32-hex trace_id
   • traceparent: W3C Trace Context (00-{trace_id}-{16-hex span_id}-{flags}) when w3c_trace_context is enabled
   • tracestate: Forwarded when valid and paired with a valid inbound traceparent
   • Configured client_ref header: Forwarded when tracing.headers.client_ref is non-empty and the inbound value passed validation

5. Retry Preservation: The same trace_id, x_request_id, and client_ref are preserved across all retry attempts, while each outbound attempt generates a fresh 16-hex span ID in traceparent.

Structured Logging¶

When tracing is enabled, log messages include trace_id alongside optional x_request_id and client_ref fields:

{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "info",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "x_request_id": "REF.001:A/7",
  "client_ref": "billing-dept-01",
  "message": "Processing chat completions request",
  "backend": "openai",
  "model": "gpt-5.6-sol"
}

W3C Trace Context¶

When w3c_trace_context is enabled, the router supports the W3C Trace Context standard:

• Incoming: Parses traceparent header (format: 00-{trace_id}-{span_id}-{flags})
• Outgoing: Generates new traceparent header with the 32-hex trace_id and a new 16-hex span ID
• State: Forwards validated tracestate header if present alongside a valid traceparent

Example traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

Disabling Tracing¶

To disable distributed tracing:

tracing:
  enabled: false

Load Balancing Strategies¶

Use the top-level selection_strategy field. The six implemented values are RoundRobin, WeightedRoundRobin, LeastLatency, Random, ConsistentHash, and PrefixAwareHash. Backend health filtering happens before selection.

selection_strategy: WeightedRoundRobin

backends:
  - name: large
    url: http://large.example.com
    weight: 3
  - name: small
    url: http://small.example.com
    weight: 1

See Load Balancing for exact semantics and current startup/hot-reload limitations.

Engine-Load-Aware Selection¶

The routing.engine_load section turns the engine_stats snapshots into a selection signal. It extends the KV overlap scorer, so it needs prefix_routing.enabled: true plus an enabled kv_cache_index for the scorer to exist, and engine_stats.enabled: true for data; continuum-router config validate warns when either dependency is missing.

routing:
  engine_load:
    enabled: false                # Scoring term switch (restart to apply)
    engine_load_weight: 0.3       # Additive scorer weight, 0.0-1.0 (restart)
    max_staleness_intervals: 3    # Snapshot freshness bound in polling
                                  # intervals, 1-100 (hot-reload, gradual)
    balance_abs_threshold: 64     # Waiting-request spread (max-min) that must
                                  # be exceeded before the term ranks (restart)
    balance_rel_threshold: 1.5    # Waiting-request ratio (max/min, >= 1.0)
                                  # that must ALSO be exceeded (restart)
    admission:
      enabled: false              # Saturation admission hint (hot-reload)
      kv_usage_threshold: 0.98    # KV usage refusal threshold, 0.0-1.0

How scoring and admission work:

• Scoring (enabled: true): Among prefix-holding candidates, the scorer favors backends reporting fewer waiting_requests (normalized by total_slots when every fresh candidate reports one, or by the candidate set's maximum waiting_requests otherwise) and lower kv_cache_usage.
• Freshness: Snapshots older than engine_stats.interval * max_staleness_intervals are treated as absent (the candidate keeps its base score, and if all candidates are stale, selection matches base scoring).
• Hysteresis dead band: Both balance_abs_threshold (max - min) and balance_rel_threshold (max / min) must be exceeded before the engine-load term ranks candidates, mirroring the SGLang Model Gateway balance gates to prevent queue oscillation from flapping selection. Gate evaluation, normalization, and the scorer cache are isolated per live candidate set. Decisions increment routing_engine_load_decisions_total{backend,reason} (see Metrics).
• Admission hint (admission.enabled: true): Evaluated at the shared selection seam independently of the scorer. When every healthy candidate has a fresh snapshot with kv_cache_usage above kv_usage_threshold, selection is refused instead of queuing. On the chat path, this refusal returns a retryable 503 that triggers fallback.fallback_chains via BackendUnhealthy; candidates without a fresh snapshot always admit.

Usage-Weighted Selection¶

The routing.usage_weight section ranks candidates by the share of a provider's declared token or request quota the router has not spent yet. Declare the provider limits on the backends themselves:

backends:
  - name: provider-a
    url: https://api.provider-a.example/v1
    provider_tpm: 200000    # Provider tokens per minute (>= 1)
    provider_rpm: 500       # Provider requests per minute (>= 1), optional

routing:
  usage_weight:
    enabled: false          # Scoring term switch (hot-reload)
    weight: 0.2             # Additive cap, 0.0-1.0
    staleness_interval: 60s # Consumption window, 1s-1h

• Declaration, not enforcement: provider_tpm and provider_rpm state what the provider allows. The router keeps dispatching past them; use rate_limiting or per-key token budgets to enforce.
• Ranking: a declared candidate scores weight x remaining_share, where remaining_share is (configured - consumed) / configured clamped to 0.0..=1.0, and the smallest share wins across the declared dimensions. A candidate that declares nothing, or has spent its whole budget, scores exactly 0.0 and keeps its pre-existing score.
• Decision: the term picks a backend only on a strict preference over every other candidate. Ties fall back to selection_strategy. Because weight is capped below the scorer threshold, the term decides sub-threshold passes that the KV or engine-load terms could not settle.
• Consumption source: router-local token metering per backend, in a tumbling window of staleness_interval. The window starts at zero, including after a restart. The RPM dimension approximates request volume from completed usage-bearing attempts.
• Metric: routing_usage_weight_decisions_total{backend,reason} with reason in applied, no_declaration, no_differentiation (see Metrics).
• Validation and warnings: weight must be finite in 0.0..=1.0, staleness_interval must parse between 1s and 1h, and a declared limit must be at least 1. config validate warns when the term is enabled with no declared limit anywhere, and when a declaration sits under a disabled term.
• Default: with the block absent or disabled, selection behavior is identical to a build without this section.

See Load Balancing for the full contract.

Upstream-Observed Quota¶

routing.quota_aware adds the upstream-observed half of quota-aware selection on top of routing.usage_weight. The router parses the quota-remaining headers a provider returns on each response and keeps the freshest snapshot per backend. While the snapshot is fresh, an observed dimension replaces the router-local estimate for that axis, so a backend with observed data participates without a declared provider_tpm or provider_rpm.

routing:
  usage_weight:
    enabled: true          # Required: quota_aware modifies this term
  quota_aware:
    enabled: true          # Capture and rank on upstream headers
    ttl: 45s               # Snapshot freshness, 1s-1h

• Headers read: Anthropic anthropic-ratelimit-requests-{limit,remaining}, anthropic-ratelimit-tokens-{limit,remaining}, anthropic-ratelimit-input-tokens-{limit,remaining}, and anthropic-ratelimit-output-tokens-{limit,remaining}; the anthropic-ratelimit-unified-{requests,tokens}-* aliases the Anthropic ingress forwards; and OpenAI x-ratelimit-limit-requests / x-ratelimit-remaining-requests plus x-ratelimit-limit-tokens / x-ratelimit-remaining-tokens.
• Normalization: a dimension is ranked only when both the remaining value and its matching -limit header are present and the limit is positive, as remaining / limit clamped to 0.0..=1.0. A remaining value without a denominator is exposed on the Admin API but never ranked, so the router-local meter stays the fallback instead of being outranked by an invented ratio. The request axis takes the provider's request share; the token axis takes the smallest usable share across tokens, input-tokens, and output-tokens.
• Capture seam: only a resolved upstream response records an observation, the same contract the router-observed TTFT follows. A transport error, connect failure, or timeout records nothing.
• Freshness: an absent or expired snapshot leaves the declared estimate in place. The default TTL matches the default engine_stats staleness window (interval 15s times max_staleness_intervals 3).
• Ownership: the headers describe the provider key, not the router backend. One snapshot is kept per backend and each response replaces it, so observations from different credentials never mix; the block is meaningful for single-credential backends.
• Visibility: GET /admin/stats/backends attaches quota beside engine_stats for each backend, and GET /admin/backends/{name}/engine-stats returns it too. The view carries the observation time, age against the live TTL, each captured dimension with its ratio, and the recognized source headers.
• Metric: routing_quota_aware_decisions_total{backend,reason} with reason in observed_used, stale, absent, unusable.
• Dependency and default: the block is inert unless routing.usage_weight.enabled is true, and config validate warns about that combination. With the block absent or disabled, selection is byte-identical to a build without it.

See Load Balancing for the full contract.

Per-Backend Retry Configuration¶

backends:
  - name: "slow-backend"
    url: "http://slow.example.com"
    retry_override:               # Complete backend-specific retry policy
      max_attempts: 5
      initial_delay: "500ms"
      max_delay: "60s"
      backoff_multiplier: 2.0
      jitter: true
      retryable_status_codes: [429, 502, 503, 504]
      retryable_errors: [ConnectionError, TimeoutError]
      timeout: "120s"

The router parses, validates, stores, and exports retry_override, but dispatch does not apply it: every backend retries under the global retry section. To stop a backend from being sent a request twice, set replay: disabled on it instead, which also covers fallback chains, the streaming relay, and quality hops; see Replay Policy.

Model Fallback¶

Configure ordered alternative models with the top-level fallback section:

fallback:
  enabled: true
  mid_stream_enabled: true
  fallback_chains:
    gpt-5.4:
      - gpt-5.4-mini
      - claude-sonnet-4-6
  fallback_policy:
    trigger_conditions:
      error_codes: [429, 500, 502, 503, 504]
      timeout: true
      connection_error: true
      model_not_found: true
      backend_unhealthy: true
    max_fallback_attempts: 3
    fallback_timeout_multiplier: 1.5
    preserve_parameters: true
    max_concurrent_dials_per_backend: 50
    content_policy_fallback: false
  empty_reply_retry:
    enabled: false       # retry a reply that carries no text at all
    max_extra_hops: 1    # 1-5
  model_settings:
    gpt-5.4:
      fallback_enabled: true
      notify_on_fallback: true

A chain model name, as a key or as an entry, is the served model id, 1 to 128 bytes of Unicode letters and digits, -, _, ., /, and :. HuggingFace paths and Ollama tags therefore chain directly, and a : that is not followed by a space needs no quotes in YAML (in TOML, quote any key that holds /, :, or .):

fallback:
  fallback_chains:
    ggml-org/gpt-oss-20b-MXFP4:
      - meta-llama/Llama-3.1-8B-Instruct
      - hf.co/org/repo:Q4_K_M

Whitespace, control characters, and other punctuation are rejected, and the error names the character and its byte offset and repeats the value escaped and capped at 128 characters. fallback.quality_check.models follows the same rule, since its entries are chain primaries. Names the operator chooses rather than the provider, such as decisions.targets[].name and model experiment names and variant ids, keep the narrower [A-Za-z0-9._-] and cannot contain /.

The regular proxy path does not produce circuit_breaker_open; use active health checks and backend_unhealthy for inference traffic. See Model Fallback for validation, headers, metrics, cross-provider translation, and the distinction between pre-stream and mid-stream behavior.

fallback.empty_reply_retry retries a non-streaming chat reply whose first choice carries no text and no tool call on the next model of the chain. It is off by default, does not trigger on a reply cut by the client's max_tokens, and is skipped for a request fallback.quality_check covers. See Empty-reply retry.

Mid-Stream Fallback¶

Mid-stream fallback allows the router to transparently continue an active SSE stream on a fallback backend when the primary backend fails mid-response. The client's connection remains open and sees an uninterrupted response with only a brief pause during the switchover.

Mid-stream fallback activates automatically when fallback.enabled: true and a fallback chain is configured for the requested model. The streaming.mid_stream_fallback section controls how the fallback backend is invoked (continuation vs restart mode), not whether fallback happens.

Disabling Mid-Stream Buffering¶

fallback.mid_stream_enabled (default true) decouples pre-stream fallback from mid-stream buffering. When set to false while fallback.enabled: true and a chain is configured:

• The initial connection still re-routes through the fallback chain on connection errors, timeouts, or trigger error codes (pre-stream fallback is unaffected).
• Once the SSE stream starts, no stream accumulator is allocated, so there is no per-stream buffering or its associated memory cost.
• A failure that happens after the stream has started surfaces to the client as a normal stream error instead of triggering a transparent switch.

Use mid_stream_enabled: false on memory-constrained hosts, or under high concurrency with many long-context streaming sessions where the ~100-200 KB per-stream buffer adds up.

This flag is distinct from streaming.mid_stream_fallback.enabled:

fallback:
  enabled: true             # Pre-stream fallback stays on
  mid_stream_enabled: false # No per-stream buffering; mid-stream errors reach the client
  fallback_chains:
    "gpt-5.6-sol":
      - "gpt-5.6-terra"
      - "gpt-5.6-luna"

Configuration¶

fallback:
  enabled: true  # Required: enables mid-stream fallback path
  fallback_chains:
    "gpt-5.6-sol":
      - "gpt-5.6-terra"
      - "gpt-5.6-luna"

streaming:
  mid_stream_fallback:
    # Enable continuation mode (default: true).
    # When true, accumulated partial response is used to build a continuation prompt,
    # producing uninterrupted output for the client.
    # When false, the fallback backend restarts the request from scratch, which may
    # cause duplicate or incoherent content if partial output was already sent.
    enabled: true

    # Minimum estimated tokens accumulated before using continuation mode (default: 50)
    # Below this threshold the request is restarted from scratch on the fallback backend
    # instead of appending a continuation prompt.
    min_accumulated_tokens: 50

    # Maximum hops after SSE output has begun (default: 2, max: 10). A hop
    # before the first chunk follows fallback.fallback_policy.max_fallback_attempts.
    max_fallback_attempts: 2

    # Prompt appended as a user message after the partial assistant response
    continuation_prompt: "Continue from where you left off exactly. Do not repeat any previously generated content."

    # Try another backend of the same model before advancing the chain
    # (default: false). See "Same-model sibling continuation" below.
    same_model_first: false

Same-model sibling continuation¶

Operators who pool one model across several backends can keep a stalled stream on that model instead of switching models. With streaming.mid_stream_fallback.same_model_first: true, a mid-stream failure first tries another backend that serves the same public model, and only advances fallback.fallback_chains when no such backend remains.

• Eligible sibling: user-routable, permitted for the API key, servable by the OpenAI-wire relay, admitted by its circuit breaker, and not already dialed for this model in this request.
• Hop semantics: the sibling receives the same continuation or restart request the relay would send to a chain target, while the response model stays the client's model. The hop counts toward max_fallback_attempts.
• Chain fallback: when the sibling set is empty, the sibling attempt also fails, or its circuit is open, the chain advances exactly as it does with the flag off.

The flag applies only to the OpenAI-wire mid-stream relay. Mid-stream recoveries emit no X-Fallback-* headers because the response headers were already sent; the hop appears as streaming_fallback_total{reason="same_model_sibling"} and in the attempt records.

How It Works¶

1. The client sends a streaming chat completion request.
2. The router begins streaming from the primary backend, accumulating response content.

3. If the backend fails mid-stream (connection drop, timeout, error event):

   • The error is NOT forwarded to the client.
   • The accumulated partial response is captured.
   • If same_model_first is on, another eligible backend serving the same model is tried first.
   • Otherwise the next healthy backend in the fallback chain is selected (unhealthy backends are skipped).
   • A continuation or restart request is sent to the fallback backend.
   • Streaming resumes on the fallback backend without closing the client connection.

4. The client receives an uninterrupted response with only a brief pause during the switchover.

Continuation vs. Restart Mode¶

The min_accumulated_tokens threshold controls which recovery mode is used:

The count compared against min_accumulated_tokens is the router's count of streamed delta events: one per content, reasoning, or tool-argument delta per chunk, which is one token per chunk on vLLM-class engines. The same count is what a stream whose backend reports no usage is charged as usage_missing.

Continuation mode (the default) produces uninterrupted output for the client. Restart mode is used automatically when there is too little context to continue meaningfully, or when the accumulated response is too long to include safely. Explicitly setting enabled: false forces restart mode unconditionally, which may cause duplicate or incoherent content visible to the client.

Edge Case Handling¶

The mid-stream fallback path addresses several edge cases automatically:

• Global timeout budget: All fallback attempts share the original request start time. Each attempt checks remaining budget before sending, preventing indefinite timeout accumulation across the chain.
• Canonical cross-provider payload: When the fallback model is on a different provider (e.g., OpenAI → Anthropic), the request stays in canonical OpenAI chat-completions form. The hop swaps the model name and nothing else; wire-format conversion, and every decision about which fields may reach the wire, happens once at dispatch, keyed on the selected backend's configured type.

• Concurrent request storms: When a primary backend goes down, all in-flight requests targeting it fail simultaneously and hop onto the same fallback backend. fallback.fallback_policy.max_concurrent_dials_per_backend (default 50, 0 = unlimited, range 0..=10000) bounds concurrent fallback dials per backend:

  • Dial-only scope: A permit is acquired just before sending a hop's outbound request and released as soon as the provider handshake returns (HTTP status and headers, or a transport error). Permits are never held across the response body, because the body phase is already bounded by server.max_concurrent_requests, request timeouts, and the circuit breaker; holding a dial permit for the full stream would turn a backend outage into an availability cliff.
  • Transport exceptions: Non-streaming hops over a Unix socket and non-streaming hops onto a Bedrock runtime backend hold the permit for the entire call because their transports do not expose a separate handshake seam.
  • Covered fallback paths: The limit applies across all four fallback paths:
    1. Non-streaming funnel (every retry attempt within a hop dials again and acquires a permit again).
    2. Pre-stream connection phase in both mid_stream_enabled modes.
    3. Hops onto native-protocol backends.
    4. Mid-stream hops performed by the relay after the SSE response is committed.
  • Primary requests excluded: Primary attempts never acquire a fallback dial permit, nor do requests whose initial backend selection already walked onto a fallback model before the first dial (when the primary is already known to be down, the fallback backend serves as the effective primary and steady-state throughput is not capped).
  • Saturation queuing and chain advancement: On a saturated backend, fallback dials queue FIFO for up to the effective timeouts.connection (and, on streaming paths, at most the remaining cross-attempt chain budget). When the wait expires, the hop fails for that backend with the backend_unhealthy trigger (enabled by default), advancing the fallback chain to the next entry or exhausting the chain with the preserved upstream failure. Native-protocol hops advance the chain on backend_unhealthy in the same way (see Model Fallback).
  • Circuit-breaker isolation: Permits are acquired before circuit-breaker admission, so a saturation timeout never consumes a half-open probe slot or records a failure on the circuit breaker.
  • Hot reload and metrics: Updating the limit immediately resets each backend's semaphore. New dials use the updated capacity while in-flight or queued dials finish on the old semaphore, briefly allowing up to old + new permits during transition. Saturation timeouts increment fallback_dial_bound_saturated_total{backend}.
  • Accumulator truncation: When accumulated response content exceeds 100 KB, the continuation mode is forced to restart to avoid sending incoherent context to the fallback backend.
  • Health re-check: Backend health is re-verified before each fallback attempt in the chain. Unhealthy backends are skipped to the next entry.
  • Missing [DONE] marker: Streams ending without [DONE] but with finish_reason: "stop" are treated as completed successfully, preventing unnecessary fallback.

Metrics¶

Three Prometheus metrics track mid-stream fallback activity. See Mid-Stream Fallback Metrics for details.

Minimizing Failover Latency¶

When a backend goes down during streaming, the time until the fallback backend takes over depends on several configuration parameters across different subsystems. Below is a tuning guide for minimizing this switchover delay.

How failover delay is composed¶

The total time a client waits during a mid-stream failover is roughly:

failover_delay ≈ failure_detection_time + health_recheck_time + fallback_connection_time

Each component maps to specific configuration:

Recommended configuration for fast failover¶

# 1. Timeouts: the most impactful settings for failover speed
timeouts:
  connection: 5s               # Faster TCP connect timeout (default: 10s)
  request:
    streaming:
      first_byte: 30s          # How long to wait for the first chunk (default: 120s)
      chunk_interval: 10s      # Max silence between later chunks before treating as failure (default: 30s)
      total: 600s              # Total streaming budget (keep generous)

# 2. Health checks: detect backend failures proactively
health_checks:
  interval: 10s                # Check every 10s instead of 30s (default: 30s)
  timeout: 3s                  # Fail health checks faster (default: 5s)
  unhealthy_threshold: 2       # Mark unhealthy after 2 failures (default: 3)
  healthy_threshold: 1         # Recover after 1 success (default: 2)
  warmup_check_interval: 1s   # Fast checks during backend startup

# 3. Fallback chain: must be configured for mid-stream fallback to activate
fallback:
  enabled: true
  fallback_chains:
    "gpt-5.6-sol":
        - "gpt-5.6-terra"
        - "gpt-5.6-luna"
  fallback_policy:
    trigger_conditions:
      error_codes: [429, 500, 502, 503, 504]
      timeout: true
      connection_error: true
      backend_unhealthy: true

# 4. Mid-stream fallback: continuation mode (default: enabled)
streaming:
  mid_stream_fallback:
    enabled: true              # Use continuation mode (default)
    max_fallback_attempts: 3   # Allow more retries for resilience (default: 2)
    min_accumulated_tokens: 30 # Lower threshold for continuation vs restart (default: 50)

Parameter impact summary¶

Failure detection scenarios¶

Different failure types are detected at different speeds:

The most common scenario in production, a backend becoming unresponsive after it has started streaming, is governed by chunk_interval. A backend that accepts the request and then goes silent is governed by first_byte. Both limits, including their per-model overrides, apply on every streaming path below: first_byte bounds the wait for the first chunk, and chunk_interval bounds every gap after it.

• The plain chat relay and the thinking-pattern relay
• The relay that serves an OpenAI-compatible client from an HTTP Anthropic backend
• The mid-stream fallback relay
• The Bedrock trait-stream pipeline and the Gemini streaming pipeline
• The Unix-socket chat, thinking-pattern, and Anthropic relays

For latency-sensitive applications, lowering chunk_interval to 10-15 seconds is recommended, with model-specific overrides for slow models:

timeouts:
  request:
    streaming:
      chunk_interval: 10s      # Fast detection for most models
    model_overrides:
      gemini-2.5-pro:          # Reasoning models need longer intervals
        streaming:
          chunk_interval: 30s
          first_byte: 120s

A chunk_interval above 60s is rejected at load time, whether it is set on streaming or in a model_overrides entry, until timeouts.limits.max_chunk_interval is raised with it; the error reports that the value is too large, maximum is 60s. The limit itself defaults to 60s and has an absolute ceiling of 300s. Every timeout ceiling is listed in Validation ceilings.

Identifying which timeout fired¶

When a streaming deadline ends a stream, the client error names the limit that fired and the value it enforced. The error type and code are unchanged (timeout on OpenAI-compatible routes, api_error on the Anthropic Messages route), so existing clients keep working.

error.details.timeout_ms carries the enforced value in milliseconds. The value is the effective one for the attempt, so it reflects per-model overrides and the scaling applied to fallback hops. On the mid-stream fallback relay, the message also names the backend (Backend '<name>' stream timed out: ...), and a fallback hop cut short by the cross-attempt budget (timeouts.streaming_fallback_budget_multiplier times total) reports fallback_budget.

{"error": {"message": "Backend stream timed out: the chunk_interval limit (30s) passed with no data from the backend", "type": "timeout", "param": null, "code": "timeout", "details": {"timeout": "chunk_interval", "timeout_ms": 30000}}}

Rate Limiting¶

The configurable, multi-dimension rate limiter is documented in Rate Limiting. Separately from it, Continuum Router has built-in, always-on protection for the model-listing endpoints, with fixed limits.

Built-in /v1/models Protection¶

GET /v1/models is rate-limited per client:

For this limiter, a client is identified by the first 16 characters of the Authorization: Bearer API key, falling back to the client IP address when no key is present. Each client has an independent quota.

POST /v1/models/refresh (force refresh) has a much tighter budget because each refresh clears the cache and fans out to every configured backend: 12 requests per minute with a burst cap of 3 per 5-second window. Only validated API keys get a per-key bucket; anonymous or invalid-token callers share a single global bucket, so spoofed headers cannot mint fresh quotas.

When a limit is exceeded, the endpoint returns 429 Too Many Requests with a message indicating whether the burst or the sustained limit was hit; rejections are also logged with the client identifier.

Cache TTL Optimization¶

To prevent cache poisoning during backend outages, empty model lists are cached for only 5 seconds initially (backing off up to 60 seconds for consecutive empty responses), while normal responses use the standard model-cache TTL.

Smart Routing¶

Smart routing classifies incoming requests by complexity and domain, then routes them to the most appropriate model tier using configurable policies. It combines a model tier registry (mapping models to tiers and domains) with a rule-based request classifier and a policy engine that maps classification results to routing decisions.

When model: "auto" is used in a chat completion request, the pipeline runs: classify the request, evaluate policies top-to-bottom, select a model from the matched tier. The same pipeline runs for all requests when intercept_all: true. Every classification call an LLM or System One classifier dispatches, a parse-failure retry included, is metered as a usage record with purpose: classifier under the classified request's key and request_id; a cached classification dispatches nothing and records nothing.

Configuration¶

smart_routing:
  enabled: true

  # Default tier when no profile matches and auto-inference is inconclusive.
  # 1 = Flagship, 2 = Standard, 3 = Lightweight. Defaults to 2.
  default_tier: 2

  # Model name that triggers smart routing. Defaults to "auto".
  virtual_model: "auto"

  # When true, all requests go through smart routing regardless of model name.
  intercept_all: false

  # Models no model_profiles entry matches: default_tier (default) keeps them
  # as candidates at their inferred or default tier; exclude leaves them out.
  unprofiled_models: default_tier

  model_profiles:
    # Exact model name
    - model: "gpt-5.6-sol"
      tier: 1
      domains: [general, code, reasoning, creative]
      cost_per_1k_input_tokens: 0.005
      cost_per_1k_output_tokens: 0.030

    # Another exact match
    - model: "gpt-5.6-terra"
      tier: 2
      domains: [general, code]
      cost_per_1k_input_tokens: 0.0025
      cost_per_1k_output_tokens: 0.015

    # Glob pattern: matches all GGUF Q4_K_M quantized models
    - model_pattern: "*-q4_K_M"
      tier: 3
      domains: [general]

  # Routing policies: first match wins, top-to-bottom evaluation
  routing_policies:
    - name: "trivial_to_lightweight"
      when:
        complexity: [trivial, simple]
        domain: [general]
      route_to:
        tier: 3

    - name: "code_to_flagship"
      when:
        domain: [code]
        complexity: [moderate, complex, expert]
      route_to:
        tier: 1
        prefer_domains: [code]

    - name: "vision_required"
      when:
        requires: [vision]
      route_to:
        tier: 1
        require_capabilities: [vision]

    - name: "complex_to_flagship"
      when:
        complexity: [complex, expert]
      route_to:
        tier: 1

    - name: "default_to_standard"
      when: {}                      # Catch-all (always matches)
      route_to:
        tier: 2

Model Profile Cost Fields¶

Both cost_per_1k_input_tokens and cost_per_1k_output_tokens are USD per 1,000 tokens. Each is optional, and each must be finite and non-negative when present: 0 is valid and means the model is free, while a negative value, .nan, .inf, and -.inf are all rejected at config load and by PUT /admin/smart-routing/model-profiles.

The rejection is a hard error, not a warning. A config.yaml carrying such a value fails at startup with a message naming the offending profile, and a hot reload keeps the previous configuration.

The same validation call also enforces a length limit: model and model_pattern must each be 200 characters or fewer, both in config.yaml and through the admin API.

Only the input cost feeds model selection. The scorer awards up to 0.3 points on cost_per_1k_input_tokens, so a lower input cost wins among otherwise equal candidates in the same tier; a profile that omits the field scores a neutral 0.15 instead. cost_per_1k_output_tokens is informational: the admin API and the WebUI serve it back, and it is validated identically, but no routing decision reads it.

Tier Classification¶

Domain Specialization Tags¶

Auto-Inference¶

When a model has no matching explicit profile or glob pattern, the router infers its tier automatically using three sources in priority order:

1. Pricing (from model-metadata.yaml, where pricing is USD per 1M tokens): input cost >= $3/1M tokens maps to Flagship; >= $0.50/1M to Standard; below that to Lightweight. Zero-cost models skip pricing inference.

2. Capabilities (from model-metadata.yaml): models with 3+ high-value capabilities (vision, reasoning, audio, video, function_calling, tool) map to Flagship; 1+ such capability or 3+ total capabilities map to Standard.

3. Name heuristics: keywords like pro, ultra, opus, sonnet, turbo map to Flagship; keywords like mini, small, tiny, nano, lite, flash, haiku and quantization markers (q4_, q5_, q8_, gguf, gptq, awq) map to Lightweight.

Auto-inferred results are cached per model ID (up to 10,000 entries). The cache clears on hot-reload and when the /admin/smart-routing/model-profiles PUT endpoint is called.

Candidate Pool and Unprofiled Models¶

Smart routing chooses its target from the healthy models the router aggregates, filtered by the same visibility rule as the user-facing model listings: a model whose every backend is internal: true or enabled: false is never a candidate, and a model that also has a public backend is checked for health on its public backends only. For a client request, per-key model/backend allow-lists and circuit admission are intersected before policy selection and rechecked at dispatch. A model advertising decisions without text or chat is never a generation candidate; a dual-capability model follows the normal eligibility gates. An internal: true backend can still serve the LLM classifier when classifier.llm.backend names it.

unprofiled_models decides what happens to a model that no model_profiles entry matches (exact model or model_pattern, local or stated by a Continuum Hub policy, or added through PUT /admin/smart-routing/model-profiles):

With exclude, a policy whose tier holds no profiled model follows the ordinary tier fallback to the other tiers. The option reloads immediately with the rest of smart_routing. continuum-router config validate warns at smart_routing.unprofiled_models when smart routing is enabled with exclude and model_profiles is empty, since every model is then excluded unless a Hub policy states profiles. GET /admin/smart-routing/status lists the pool as candidate_models (each with model, tier, and source: explicit_exact, explicit_pattern, auto_inferred, or default), unprofiled exclusions as excluded_unprofiled_models, and decision-only output-contract exclusions as excluded_incompatible_models.

Glob Pattern Syntax¶

Patterns use * as the only wildcard character. Multiple wildcards are supported.

Request Classifier¶

The rule-based classifier analyzes each request using 11 signal types and produces a ClassificationResult containing complexity level, domain tag, required capabilities, and confidence score.

Complexity Levels¶

The expert Ceiling of the Rule Classifier¶

The rule-based classifier cannot recognize most expert requests unless they are long. Its complexity score is a weighted average of per-signal ratios, bucketed at 0.15, 0.35, 0.55, and 0.75, and expert needs 0.75 or more. An average never exceeds its strongest term, and most terms stop below 0.75. The table lists each term's weight and maximum ratio:

The classifier scores only the current turn (see What the Rule Classifier Scores), so the best case in each length band works out the same for any request:

• Under 200 estimated tokens (the system and developer messages plus the latest user message) without math notation, the score cannot exceed (0.4*0.15 + 0.8*0.25 + 0.75*0.2) / 0.6 = 0.683.
• Under 50 estimated tokens with every term at its maximum, including math notation, it cannot exceed (0.25*0.15 + 0.8*0.25 + 0.75*0.2 + 1.0*0.2) / 0.8 = 0.734.
• System prompts and tool definitions add terms whose maximum sits below 0.75, so they pull an agentic request down rather than up. Earlier turns of the conversation add nothing.

Without math notation, the rule classifier reaches expert only for requests over about 500 estimated tokens. With math notation, there is one narrow non-proof path for shorter prompts: a request of 51 to 200 estimated tokens with math notation and four or more distinct complex keywords scores (0.4*0.15 + 0.8*0.25 + 1.0*0.2) / 0.6 = 0.767 (0.7625 with a fenced code block; three keywords top out at 0.747). If the same request includes inline code or code_intent, the added 0.5-ratio term pulls the average back below 0.75. Short, difficult design prompts (such as a novel architecture or a research-level system design question) therefore score complex at most, and usually moderate. On the accuracy dataset, a Byzantine consensus protocol design (labelled expert) and a rate-limiter design (labelled complex) produce identical signal vectors, so weight tuning alone cannot separate them.

Formal Proof Exception (expert Floor)¶

Formal proofs are the one exception that reaches expert regardless of length. A request receives the expert floor when the latest user message contains both math notation (a LaTeX macro or two math delimiters) and proof wording, provided the user is asking for a proof rather than about one.

If reject_expert: true is configured for a load state under smart_routing.load_management.degradation (default false), short proof requests classified as expert are rejected in that load state.

Impact by Classifier Method and Validation Warnings¶

The expert ceiling applies wherever routing relies on the rule classifier's output:

custom_complexity_levels does not change this behavior because only the LLM and System One classifiers emit custom complexity levels.

continuum-router config validate warns at smart_routing.routing_policies[N].when.complexity when a policy matches [expert] without also matching complex:

• When the warning fires: Smart routing is enabled and either method is rule, hybrid (with rule.confidence_threshold <= 0.95), or hybrid_system_one (with rule.confidence_threshold <= 0.95), or method is llm/hybrid with no resolved classifier.llm.backend (where all requests fall back to the rule classifier).
• When the warning is suppressed: The policy lists [complex, expert], or a hybrid method sets rule.confidence_threshold > 0.95 (above the rule classifier's 0.95 ceiling, causing every request to escalate).
• Recommendation: To route difficult prompts by difficulty rather than length, include both [complex, expert] in the policy or use method: llm or method: system_one.

What the Rule Classifier Scores¶

The rule classifier scores the request being made, not the conversation: the system and developer messages plus the latest user message. Earlier user and assistant turns, and anything after the latest user message (an assistant prefill, tool results), are not read for complexity, domain, or confidence. This is the same text the LLM classifier judges, so hybrid decides whether to escalate on what the escalation would read, and rule, hybrid, and every rule fallback under llm, system_one, and hybrid_system_one share one rule result. A context-dependent follow-up ("now do the same for the whole file") is classified by its own words, which is also how the LLM and System One classifiers read it.

Only required_capabilities reads the whole conversation, because an image in an earlier turn still needs a vision backend and tools is a request-level field. The published language, which when.language policies match and which selects the keyword table, is the latest user message's own language, so a Korean question after an English conversation routes as a Korean question; when that message has no letters to detect from (??), the language of every user turn is used instead.

Classification Signals¶

Each detected signal contributes to the final complexity level and domain tag. Conflicting signals (e.g., both creative and code markers present) reduce the confidence score.

The domain is decided by the strongest intent signal (code_blocks, inline_code, math_notation, creative_markers, analysis_markers, code_intent, translation_intent). multilingual is a script observation rather than an intent, so it determines the domain only when no intent signal fires:

• translation_intent and multilingual (strength 0.72):
  • Maps to the multilingual domain, matching what the LLM and System One classifiers return for translation tasks. when.domain: [multilingual] matches translation requests in English, Korean, Japanese, or Chinese on any deployment, and primary_language does not suppress translation_intent.
  • Ranks above a single creative marker or a two-marker analysis match (so "translate this poem" routes as multilingual), and below code_intent (0.75), math_notation (0.8), and fenced code blocks (0.9) (so translating fenced code, prompts with two code markers, or formulas stays code or reasoning).
  • Counts toward rule confidence like any intent signal, allowing short translation requests to clear the default 0.7 hybrid gate.
  • Accepted over-reads: Idiomatic uses ("this translates to a 2x speedup"), topical mentions ("기계 번역 품질을 분석해줘"), and unfenced code translation requests with only a single code marker ("Translate this Python function to Rust.") resolve to multilingual. Non-primary-script text with no intent signal also falls back to multilingual.
• code_intent (strength 0.75):
  • Detects requests to write code when no literal code markup is present (for example, "Implement the rate limiting logic for this REST API.").
  • Requires two distinct entries from the code keyword list to match (pairing an action and an object prevents single verbs like "Write a poem about autumn leaves" or single nouns like "What is a REST API?" from misclassifying as code).
  • Skipped completely when the request already contains a fenced code block or inline backticks, leaving domain and confidence unchanged.
  • Ranks below math_notation (0.8) so LaTeX-heavy requests stay reasoning, and above multilingual (0.7) so unmarked Korean code prompts do not fall back to script observation.

Language Awareness¶

The classifier detects the language of the user's own message text and publishes it on the classification result. Detection is script-based: Hangul means ko, kana means ja, Han without kana means zh, Latin script is reported as en, and anything else is und (undetermined). A tenth of the letters is enough for Hangul or kana, so a Korean or Japanese request that quotes English technical vocabulary is still detected as Korean or Japanese.

The detected language selects the keyword table the complexity, creative, and analysis signals match against. English and Korean ship built in, and every table includes the English keywords, so a request that mixes languages keeps firing the signals it would have fired on the English list alone. A language with no table of its own falls back to English.

Request length is estimated from characters, weighted by script (a Hangul syllable or a CJK ideograph counts as two and a half Latin characters), not from UTF-8 bytes. The weight is chosen so that a Korean request and its English translation estimate the same length on average: on the repository's paired English/Korean fixtures the Korean members estimated 12.9% shorter at two Latin characters and 1.6% longer at two and a half. A single short pair can still straddle a message_length step when one wording carries an extra word.

Primary language¶

classifier.rule.primary_language names the language this deployment's traffic is normally written in, as a short BCP-47 tag (ko, ko-KR, ja, fr; case-insensitive, only the primary subtag is read).

Setting it suppresses the multilingual signal for requests in that language. The signal fires on any text that is mostly non-ASCII, so on a Korean deployment it fired on every ordinary Korean request, and a Korean greeting (which carries no intent signal at all) was classified multilingual where its English translation was classified general. Naming the primary language says that text in that language is not itself a translation task, and such a request is then classified on its intent alone. Requests in any other language keep the signal.

It also selects the keyword table for Latin-script text, since detection reports every Latin-script language as en. primary_language: fr is what makes a keywords entry for fr reachable. A tag detection recognizes on its own (ko, ja, zh) never overrides detection: Latin-script text on a Korean deployment is still matched against the English table.

The setting is optional. Left unset, every language keeps the multilingual signal.

Custom keywords¶

classifier.rule.keywords adds keywords per language. Entries are merged into the built-in tables rather than replacing them, every language inherits the English built-ins, and values are trimmed and lowercased with duplicates dropped. A tag with no shipped table (fr, de) creates a new table on top of the English base, activated by setting primary_language to the same tag.

Keyword matching rules:

• Single-word keywords: Matched as plain substrings (for example, the Korean stem 요약 matches inside 요약해줘 and poem matches inside poems).
• Multi-word keywords: Match when all words appear in order within a single sentence with at most two tokens between consecutive words (so write a story matches "write a short story" and 시를 써 matches "시를 하나 써줘"). Sentence terminators (., !, ?, or a newline) close the matching window.
• Token boundaries in multi-word keywords: Every word must match a whole token (ignoring edge punctuation), except the last word when it ends in a non-Latin character, which may be followed by trailing characters in the same token (so 써 in 시를 써 matches 써줘, while leading 시 in 시 한 편 never matches 시스템). Latin-script words always require full-token matches.
• Adjacency requirement for Latin function-word idioms: When both the first and last words of a multi-word marker are Latin function words (a, the, is, what, once, upon, etc.), the two-token gap allowance is disabled so idioms like once upon or what is do not false-trigger across unrelated words ("Once agreed upon...", "What throughput is..."). Markers anchored by content words (write a story) keep gap tolerance.
• Compound-noun trade-off: Gap tolerance does not distinguish modifiers from compound nouns ("Write a short story" and "Write a user story" both match write a story and classify as creative). If a multi-word marker causes misrouting on domain-specific compounds, replace it with narrower keywords.

Built-in Korean keyword disambiguation:

To match the request-phrase semantics of the English keyword table rather than bare question words or nouns, the built-in Korean keywords apply morphological guards:

smart_routing:
  classifier:
    rule:
      # The language this deployment's traffic is written in. Suppresses the
      # `multilingual` domain signal for that language. Optional.
      primary_language: ko

      # Extra keywords, merged into the built-in tables. Optional. A `fr`
      # entry here would need `primary_language: fr` to be reachable.
      keywords:
        ko:
          complex: ["커널 스케줄러"]
          creative: ["판소리 사설"]
          # `code` fires only when two distinct entries match, so add
          # entries in pairs: an action and an object.
          code: ["배포", "헬름 차트"]

Both fields hot-reload: a change rebuilds the rule classifier in place.

LLM-Based Classifier¶

The rule-based classifier is fast but sometimes ambiguous. When that is not accurate enough, the LLM-based classifier sends the request to a small, cheap model for classification, and the System One classifier (below) asks a decision model a typed battery instead. Five operating modes are supported via classifier.method:

Hybrid mode behavior and escalation characteristics:

• Threshold gate: hybrid routes on the rule result when its confidence reaches rule.confidence_threshold (default 0.7) and escalates to the LLM classifier otherwise.
• Signal-count confidence: Because rule confidence counts how many signals fired rather than estimating label accuracy, escalation volume depends on signal density rather than prompt ambiguity. On the labelled accuracy dataset, hybrid escalates 53.7% of requests (complexity AUROC 0.46, ranking right vs. wrong labels no better than chance), and on public English and Korean benchmark prompts 91% to 99% fall below 0.7 and pay the LLM round trip.
• Monitoring and alternatives:
  • Check actual escalation rates on your workload via smart_routing_hybrid_gate_total (see Metrics).
  • To evaluate a System One decision model without LLM round trips, run method: rule with shadow: true (note that requests still wait for the shadow call up to timeout_ms).
  • hybrid_system_one places a decision model with a calibrated min_confidence behind the same rule threshold at decision-model latency (47 ms to 291 ms p50 measured; see the system_one table below), escalating to the LLM classifier only when the decision model's confidence falls below min_confidence. On mostly Korean traffic, System One has not outperformed the rule classifier (see Decision Models).

smart_routing:
  enabled: true

  classifier:
    # Classification method: "rule" (default), "llm", or "hybrid".
    method: hybrid

    rule:
      # Confidence below this threshold triggers LLM escalation in hybrid mode.
      # Range: 0.0 – 1.0. Default: 0.7.
      confidence_threshold: 0.7

      # Language this deployment's traffic is written in, as a short BCP-47
      # tag. Suppresses the `multilingual` domain signal for that language.
      # Optional; see "Language Awareness" above.
      primary_language: ko

      # Extra classification keywords per language, added to the built-in
      # tables. Optional.
      keywords:
        ko:
          creative: ["판소리 사설"]
          # `code` fires only when two distinct entries match, so add
          # entries in pairs: an action and an object.
          code: ["배포", "헬름 차트"]

    llm:
      # Model used for classification. Any fast, cheap model works well.
      model: "gpt-5.6-terra"

      # Backend name to route classification requests to. Must be a configured
      # backend. If omitted, the router uses the first backend that ordinary
      # user traffic can reach, skipping any `internal: true` or
      # `enabled: false` entry. See "Classifier Backend Selection" below.
      backend: "openai-fast"

      # Maximum time allowed for a classification request (milliseconds).
      timeout_ms: 2000

      # Maximum input tokens sent to the classifier (content is truncated).
      max_input_tokens: 500

      # Number of retries after a parse failure (0 or 1). Default: 1.
      max_retries: 1

      # Temperature for classification. 0.0 gives deterministic output.
      # Range: 0.0 – 2.0. Default: 0.0.
      temperature: 0.0

      # Maximum output tokens in the classification response.
      max_output_tokens: 150

      # Structured output strategy. "auto" selects the best method for the
      # configured backend: json_schema (OpenAI/vLLM), tool_use (Anthropic),
      # json_object (Ollama/Gemini/LM Studio), prompt_only (others).
      structured_output: auto   # auto | json_schema | tool_use | json_object | prompt_only

      # Include built-in few-shot examples in the system prompt.
      few_shot_examples: true

      # Custom few-shot examples appended after the built-in ones.
      custom_examples:
        - user: "What is the recommended dose of ibuprofen?"
          classification:
            complexity: simple
            domain: medical

      # Cache TTL for classification results (seconds). Default: 300.
      cache_ttl_seconds: 300

      # Maximum number of cached entries. Default: 10000.
      max_cache_entries: 10000

      # Disable the LLM classifier when load reaches this state.
      # "critical" (default) or "warning". Set to "" to never disable.
      disable_under_load_state: critical

    # Extend the built-in complexity taxonomy with custom levels.
    custom_complexity_levels:
      - name: specialized
        description: "Requires a domain-specific fine-tuned model"
        rank: 6   # Optional ordering hint (higher = harder)

    # Extend the built-in domain taxonomy with custom categories.
    custom_domains:
      - name: medical
        description: "Medical and clinical questions"

rule.confidence_threshold and llm.temperature are each validated at config load: confidence_threshold is bounded to [0.0, 1.0] inclusive, and temperature to [0.0, 2.0] inclusive, matching the range the router already enforces for the canonical temperature request parameter elsewhere in this document. Both bounds are inclusive, so 0.0 and the upper bound are legitimate settings, not accidents: confidence_threshold: 0.0 makes hybrid mode always escalate to the LLM classifier, and confidence_threshold: 1.0 makes it never escalate. An out-of-range or non-finite value (.nan, .inf, -.inf) is rejected the same way the model profile cost fields above are: a config.yaml carrying such a value fails at startup with a message naming the field, and a hot reload keeps the previously loaded configuration.

When classifier.backend (or the classifier's resolved backend URL) points at a native Anthropic backend and classifier.llm.model is a Claude Opus or Sonnet 4.7 or later, the configured temperature is silently dropped from the classification request rather than sent and rejected, the same sampling-parameter restriction described for the main request path in Anthropic Extended Thinking Models.

The classifier dispatches its classification request through the same backend implementation that serves normal traffic, so every configured backend type is supported and uses that backend's own authentication and request transformation. If classifier.llm.backend names a backend that is not in the live backend pool, classification fails for that call with an error naming the backend and falls back to the rule-based classifier, rather than failing at startup.

Classifier Backend Selection¶

Classification sends the user's own prompt text to the classifier backend, so which backend gets picked decides where that content goes. The choice is made once, when the router builds its state and again on each hot reload, not per request, so a given configuration keeps sending classification prompts to the same backend until the configuration changes.

The guardrail input stage runs before the LLM or System One classifier (and before a System One shadow): an enforce-mode block stops the request before the classifier is called, and a transform verdict such as PII masking masks every message where it lives, the system prompt included, before the classifier reads it, or refuses the request when the mask cannot be applied that way. In monitor mode the request is never altered, so the classifier receives the prompt as sent. See Guardrails: lifecycle hooks for the pass order and when a request is evaluated twice.

classifier.llm.backend and its absence are resolved differently on purpose:

The name is trimmed, and a blank value is read as omitted, the same way at load and when the classifier is built. config validate and the router resolve the backend with the same function, so the warning appears exactly when the router would run rule-based. A Continuum Hub statement that pins a classifier backend overrides the local section; hybrid_system_one is not warned about, because its LLM step is an optional last escalation after System One.

System One Classifier¶

A System One decision model (TypeSafe's hosted Jev, Microsoft-Decision-1 on Foundry, or a native llama.cpp decision model) answers typed questions over a state in one forward pass and returns provider-reported probabilities and, where available, confidence. The System One classifier asks two questions per request, a score over the ordered complexity levels and a choice over the domains, and fills the same ClassificationResult the other classifiers produce, so routing_policies[].when.complexity and .domain need no change. What changes is the cost of an uncached classification and the meaning of confidence:

The shared System One target block (protocol, auth, endpoint + api_key_env or an eligible backend, plus model, provider bounds and the existing timeout/budget/concurrency fields) is flattened into classifier.system_one; see Decision Models: Provider profiles and experimental media for the complete target contract. The classifier adds its own fields:

smart_routing:
  classifier:
    method: hybrid_system_one   # or system_one; rule | llm | hybrid may add shadow: true
    rule:
      confidence_threshold: 0.7 # hybrid_system_one: ask System One below this

    system_one:
      # Hosted Jev:
      protocol: typesafe
      endpoint: "https://api.typesafe.ai/v1/systemone"
      api_key_env: TYPESAFE_API_KEY
      model: jev-1.14.0          # Pin the version used for calibration.
      # Or a self-hosted server already configured as a backend (mutually
      # exclusive with endpoint; internal: true backends are allowed):
      # backend: laya
      timeout_ms: 1500
      max_concurrent: 32

      # A System One answer below this confidence is not routed on. Range
      # 0.0-1.0, default 0.15 (measured; see Decision Models).
      min_confidence: 0.15

      # Run the battery beside the routing classifier without changing any
      # decision; see "Shadow mode" below. Default false. Rejected with
      # method system_one / hybrid_system_one.
      shadow: false

      # Optional noul questions, recorded as classification signals only.
      signals:
        needs_tools: false      # system_one_needs_tools (capability signal)
        is_sensitive: false     # system_one_is_sensitive (domain signal)

      max_input_tokens: 500     # Truncation of the last user request.
      cache_ttl_seconds: 300
      max_cache_entries: 10000
      disable_under_load_state: critical   # critical | warning | ""

    # Optional last escalation step for hybrid_system_one.
    llm:
      model: "gpt-5.6-terra"
      backend: openai-fast

Target Protocols and Rollout Workflow¶

The System One classifier always dispatches native decision calls; it never uses the public Decisions API's chat-model emulation path:

• TypeSafe (protocol: typesafe): Point endpoint at https://api.typesafe.ai/v1/systemone with api_key_env (for hosted Jev) or set backend to an eligible self-hosted TypeSafe-compatible backend (such as laya).
• Microsoft Foundry (protocol: foundry): Point endpoint at a Foundry resource root or Microsoft provider endpoint, set auth: api_key (or entra for an externally refreshed token), and set model to your Foundry deployment name.
• Native llama.cpp (protocol: llama_cpp): Point endpoint at the server's /v1/systemone URL (or set backend to an eligible self-hosted backend) and set model to its configured model alias.

When onboarding a new decision target:

1. Start with method: rule and system_one.shadow: true.
2. Measure agreement and confidence distributions on representative English and Korean traffic.
3. Run make calibrate-systemone against the exact model, version, and question battery.
4. Switch method to system_one or hybrid_system_one using the calibrated min_confidence threshold, and remove shadow: true.

The default min_confidence: 0.15 is the measured reference value for self-hosted Laya and does not transfer to Jev, Microsoft-Decision-1, Kev, Decision 2/D3, or other checkpoints.

Question Battery¶

The classifier sends a two-question routing battery (plus up to two optional diagnostic noul questions) on every call:

The battery asks only about the request's own difficulty and domain, never about a candidate model's expected success.

Confidence and Fallback Behavior¶

• Combined confidence: The classification confidence is the minimum of the complexity and domain confidences, requiring both answers to be concentrated before the gate trusts the result.
• Low-confidence fallback:
  • Under method: system_one, an answer below min_confidence is discarded in favor of the rule classifier result.
  • Under method: hybrid_system_one, an answer below min_confidence escalates to the LLM classifier when classifier.llm resolves a backend (see Classifier Backend Selection), or falls back to the rule result otherwise.
• Error and load fallback: Any failed call, malformed response, unresolved target, or load-state disablement returns the rule result and increments smart_routing_system_one_classifier_fallbacks_total{reason} (error, low_confidence, or load).
• Capabilities and language: required_capabilities and detected language always come from the rule classifier (both decision-model vendors report weaker CJK accuracy, and rule detection is free).

Request State, Privacy, and Cache Key¶

• Outbound state payload: Built from the exact same fields as the LLM classifier's input:
  • request: The latest user message, truncated to max_input_tokens (at 4 characters per token).
  • system_excerpt: The first 200 characters of the system or developer message (included only when present).
  • message_count, has_images, has_tools, and model (the requested model).
  • Earlier conversation turns and public Decisions API media are never sent.
• Guardrail ordering and third-party privacy: The state is constructed after the guardrail input stage (including when shadow: true). Prompts blocked by an enforce guardrail never reach the classifier, and prompts modified by a transform guardrail (such as PII masking) are sent masked (system_excerpt included); in monitor mode the prompt is sent as received. Because hosted Jev and Microsoft Foundry are third-party services, privacy-sensitive deployments should point endpoint or backend at a self-hosted server.
• Cache key isolation: The SHA-256 cache key hashes the bounded post-guardrail state together with the target profile, requested decision-model identity, cache-format version, generated question battery, and (for backend: targets) the live configuration-snapshot topology generation. Reloading the target, profile, credentials/headers, disabled state, model, or battery invalidates prior entries. Credential values and prompt text never appear in cache keys or metric labels.

Usage Accounting and Metrics¶

• Per-attempt usage records: Every native wire attempt (including failures, retries, and shadow calls) is persisted and exported as a separate usage record with purpose: classifier, the resolved response model, target/backend attribution, input/output token counts, latency, and outcome. Cache hits dispatch no call and emit no usage record.
• GET /admin/stats separation: Classifier calls and tokens are reported under by_purpose.classifier and excluded from client request counts (by_purpose.client), preventing both omission and double-counting.
• Edge-case accounting:
  • A malformed HTTP-2xx decision response is recorded as upstream_error with HTTP status 200 and zero output tokens.
  • A valid 2xx response whose answer omits confidence preserves its billed token usage before taking the routing fallback.
  • A successful response retains nonzero output token usage even when the model's catalog output price is 0; cost accounting applies the version-specific catalog price to the retained tokens rather than zeroing out token counts.
• Prometheus metrics: Exports smart_routing_system_one_classifier_input_tokens_total{counted}, systemone_requests_total{consumer="classifier", result}, systemone_request_duration_seconds{consumer="classifier"}, and smart_routing_system_one_classifier_cache_hits_total.

Shadow Mode and Measured Laya Reference¶

Set shadow: true with method: rule, llm, or hybrid to evaluate a System One model on live traffic without changing routing decisions:

• Execution and metrics: The System One battery runs concurrently with the active routing classifier on every request. Routing uses the active classifier's result, while field-by-field agreement is recorded in smart_routing_shadow_agreement_total{field, agree} (field is complexity or domain, agree is true or false). When debug_headers: true, responses also include X-Smart-Route-Shadow: complexity=<level>;domain=<tag>;confidence=<0.00>;agree=<both|complexity|domain|none>.
• Latency and load: Because the request waits for both classifiers to finish, running shadow: true alongside method: rule adds the decision call latency to every request (up to timeout_ms, default 1500 ms), whereas alongside llm or hybrid the two network calls overlap. Keep timeout_ms low during shadow evaluation. Shadow calls respect disable_under_load_state, shed load before serving degrades, and log failed wire attempts at debug.
• Measured Laya benchmark (min_confidence: 0.15): Derived on 2026-09-24 against the compose file's self-hosted laya-serve (CPU) on the routing corpus:
  • Overall accuracy (domain + complexity): 0.510 at min_confidence: 0.15, compared to 0.460 at 0.5 and 0.475 for the rule classifier alone.
  • Korean slice accuracy: 0.405 for Laya vs. 0.487 for the rule classifier. Deployments with predominantly Korean traffic should remain on method: rule (with shadow: true while testing) until their target model demonstrates higher accuracy. See Decision Models for per-slice tables and the make calibrate-systemone workflow.

Validation, Hot Reload, and Recursion Guard¶

• Load-time validation: The router validates the shared target rules (exactly one of endpoint or backend, the /v1/systemone path, and backend eligibility/visibility), min_confidence finite in [0.0, 1.0], disable_under_load_state in ["critical", "warning", ""], presence of system_one when method is system_one or hybrid_system_one, absence of shadow: true when method is system_one or hybrid_system_one, at most 5 custom_complexity_levels, and classifier.llm.backend validity when an LLM escalation path is active.
• Hot reload: All system_one fields hot-reload with smart_routing. Switching between method: llm and method: system_one rebuilds only the required classifiers, and backend: targets read the live backend configuration on every call so Admin API or config-sync updates apply immediately.
• Recursion guard: Decision calls target /v1/systemone directly and never pass through the router's /v1/chat/completions pipeline, preventing self-classification loops. LLM escalation calls in hybrid_system_one continue to carry X-Smart-Route-Bypass: true (see Bypass Header).

Structured Output Strategies¶

The LLM classifier needs structured JSON from the classifier model. The auto strategy picks the right mechanism based on the backend type, but you can override it explicitly:

When the classifier response cannot be parsed, the router retries once with a correction prompt (controlled by max_retries). If the retry also fails, the result from the rule-based classifier is used instead.

Classification Cache¶

Classification results are cached in memory with a configurable TTL (cache_ttl_seconds) and entry bound (max_cache_entries) to avoid repeated upstream calls for the same request. For the LLM classifier, the cache key is a SHA-256 hash of the truncated user message. For the System One classifier, the cache key hashes the cache-format version, target profile, model, generated question battery, live cache_target_identity (including the live configuration-snapshot generation for backend: targets), and the bounded post-guardrail state (request, system_excerpt, message_count, has_images, has_tools, model), so any reload of target, profile, credentials or headers, disabled state, model, or battery invalidates cached results. The cache is per-process and not shared across router instances.

Custom Taxonomy¶

Both complexity levels and domain tags are extensible. Custom values added via custom_complexity_levels and custom_domains appear in the classifier's system prompt and are accepted in the structured-output schema. Routing policies can reference custom values just like built-in ones:

routing_policies:
  - name: specialized_to_flagship
    when:
      complexity: [specialized]
    route_to:
      tier: 1

Bypass Header¶

The LLM classifier sends an X-Smart-Route-Bypass: true header with its classification requests. The router skips smart routing for any request carrying this header, preventing circular classification loops when the classifier backend is itself behind the same router instance.

Routing Policies¶

Policies are evaluated top-to-bottom; the first match wins. If no policy matches and no catch-all is defined, the request falls back to default_tier.

Policy Condition Logic¶

• Fields within a when block are AND-ed: all specified fields must match.
• Values within a single field are OR-ed: complexity: [trivial, simple] matches either.
• when: {} is a catch-all that always matches. A condition that sets only an identity field (key_tier, org, language, header) or only input_tokens is not a catch-all.

Policy Fields¶

when conditions:

Identity Conditions¶

The first three fields describe what the request is. The last four describe who sent it and how they asked for it to be handled, so one auto alias can route differently per key, per organization, per request language, or on an explicit client hint.

routing_policies:
  # A paying tier gets the flagship models.
  - name: "build_tier_flagship"
    when:
      key_tier: [tier_build]
    route_to:
      tier: 1

  # One organization is pinned to the cheapest tier regardless of complexity.
  - name: "sandbox_org_lightweight"
    when:
      org: [org_sandbox]
    route_to:
      tier: 3

  # Korean requests go to a model that handles them well.
  - name: "korean_to_flagship"
    when:
      language: [ko]
    route_to:
      tier: 1

  # A client can ask for a more thorough answer.
  - name: "thorough_hint"
    when:
      header:
        x-route-hint: [thorough]
    route_to:
      tier: 1

  - name: "default_to_standard"
    when: {}
    route_to:
      tier: 2

key_tier matches the Continuum Hub tier id (tier_id) resolved for the request's API key. A binary built without the control-plane feature never resolves a tier, and neither does a request whose key is not hub-bound, so a policy naming key_tier simply never matches there. It never matches everything: a tier-gated policy fails closed rather than opening up where tiers do not exist. continuum-router config validate warns when a policy uses key_tier on a build that cannot resolve one.

org matches the organization_id of the authenticated API key. An unauthenticated request carries no organization and never matches.

language reads the language the classifier detected in the user's own message text, the same detection that drives the language-aware keyword tables. Only the primary subtag is significant and the comparison is case-insensitive, so ko, KO, and ko-KR all name Korean. Two absence cases differ deliberately: a request with no letters at all (an empty message, or digits and punctuation only) carries no language and matches no language clause, while text that was examined but could not be attributed carries und and matches a clause that lists und.

header reads request headers. Names and values are both matched case-insensitively, and an entry with an empty value list is a presence check (x-route-hint: [] matches the header with any value). Values are bounded at 256 bytes; a longer value is treated as absent rather than truncated, so a client cannot match a policy by sending a long value that merely starts with the configured one. Because a client sets these headers itself, use header to express a caller's preference and gate anything privileged on key_tier or org, which a client cannot set. Header values never appear in a metric label or a log line.

When smart_routing.debug_headers is on, x-smart-route-policy names the policy that matched, which is the quickest way to confirm an identity condition is firing. POST /admin/smart-routing/simulate also accepts key_tier, org, and headers fields alongside payload, so an operator can check a tier-gated policy without holding a key on that tier.

Request Size Condition¶

input_tokens routes on the estimated size of the request, so a long prompt reaches a long-context tier on the first attempt instead of failing on a small-context model with 400 context_length_exceeded and relying on fallback.

routing_policies:
  # Long conversations go to the long-context tier.
  - name: "long_context"
    when:
      input_tokens: { gt: 120000 }
    route_to:
      tier: 1

  - name: "default_to_standard"
    when: {}
    route_to:
      tier: 2