Advanced Configuration¶

Global Prompts¶

Global prompts allow you to inject system prompts into all requests, providing centralized policy management for security, compliance, and behavioral guidelines. Prompts can be defined inline or loaded from external Markdown files.

Basic Configuration¶

global_prompts:
  # Inline default prompt
  default: |
    You must follow company security policies.
    Never reveal internal system details.
    Be helpful and professional.

  # Merge strategy: prepend (default), append, or replace
  merge_strategy: prepend

  # Custom separator between global and user prompts
  separator: "\n\n---\n\n"

External Prompt Files¶

For complex prompts, you can load content from external Markdown files. This provides: - Better editing experience with syntax highlighting - Version control without config file noise - Hot-reload support for prompt updates

global_prompts:
  # Directory containing prompt files (relative to config directory)
  prompts_dir: "./prompts"

  # Load default prompt from file
  default_file: "system.md"

  # Backend-specific prompts from files
  backends:
    anthropic:
      prompt_file: "anthropic-system.md"
    openai:
      prompt_file: "openai-system.md"

  # Model-specific prompts from files
  models:
    gpt-4o:
      prompt_file: "gpt4o-system.md"
    claude-3-opus:
      prompt_file: "claude-opus-system.md"

  merge_strategy: prepend

Prompt Resolution Priority¶

When determining which prompt to use for a request:

1. Model-specific prompt (highest priority) - global_prompts.models.<model-id>
2. Backend-specific prompt - global_prompts.backends.<backend-name>
3. Default prompt - global_prompts.default or global_prompts.default_file

For each level, if both prompt (inline) and prompt_file are specified, prompt_file takes precedence.

Merge Strategies¶

REST API Management¶

Prompt files can be managed at runtime via the Admin API:

# List all prompts
curl http://localhost:8080/admin/config/prompts

# Get specific prompt file
curl http://localhost:8080/admin/config/prompts/prompts/system.md

# Update prompt file
curl -X PUT http://localhost:8080/admin/config/prompts/prompts/system.md \
  -H "Content-Type: application/json" \
  -d '{"content": "# Updated System Prompt\n\nNew content here."}'

# Reload all prompt files from disk
curl -X POST http://localhost:8080/admin/config/prompts/reload

See Admin REST API Reference for complete API documentation.

Security Considerations¶

• Path Traversal Protection: All file paths are validated to prevent directory traversal attacks
• File Size Limits: Individual files limited to 1MB, total cache limited to 50MB
• Relative Paths Only: Prompt files must be within the configured prompts_dir or config directory
• Sandboxed Access: Files outside the allowed directory are rejected

Hot Reload¶

Global prompts support immediate hot-reload. Changes to prompt configuration or files take effect on the next request without server restart.

Model Metadata¶

Continuum Router supports rich model metadata to provide detailed information about model capabilities, pricing, and limits. This metadata is returned in /v1/models API responses and can be used by clients to make informed model selection decisions.

Metadata Sources¶

Model metadata can be configured in three ways (in priority order):

1. Backend-specific model_configs (highest priority)
2. External metadata file (model-metadata.yaml)
3. No metadata (models work without metadata)

External Metadata File¶

Create a model-metadata.yaml file:

models:
  - id: "gpt-4"
    aliases:                    # Alternative IDs that share this metadata
      - "gpt-4-0125-preview"
      - "gpt-4-turbo-preview"
      - "gpt-4-vision-preview"
    metadata:
      display_name: "GPT-4"
      summary: "Most capable GPT-4 model for complex tasks"
      capabilities: ["text", "image", "function_calling"]
      knowledge_cutoff: "2024-04"
      pricing:
        input_tokens: 0.03   # Per 1000 tokens
        output_tokens: 0.06  # Per 1000 tokens
      limits:
        context_window: 128000
        max_output: 4096

  - id: "llama-3-70b"
    aliases:                    # Different quantizations of the same model
      - "llama-3-70b-instruct"
      - "llama-3-70b-chat"
      - "llama-3-70b-q4"
      - "llama-3-70b-q8"
    metadata:
      display_name: "Llama 3 70B"
      summary: "Open-source model with strong performance"
      capabilities: ["text", "code"]
      knowledge_cutoff: "2023-12"
      pricing:
        input_tokens: 0.001
        output_tokens: 0.002
      limits:
        context_window: 8192
        max_output: 2048

Reference it in your config:

model_metadata_file: "model-metadata.yaml"

model_metadata_file is only the base layer. The router also assembles metadata from model-metadata.d/ drop-in directories and a model_metadata_dirs config key, so an operator's overrides do not have to live in the file metadata download overwrites. See Layered model metadata for the search order, the merge rules, and metadata show.

Thinking Pattern Configuration¶

Some models output reasoning/thinking content in non-standard ways. The router supports configuring thinking patterns per model to properly transform streaming responses.

Pattern Types:

Configuration Example:

models:
  - id: nemotron-3-nano
    metadata:
      display_name: "Nemotron 3 Nano"
      capabilities: ["chat", "reasoning"]
      # Thinking pattern configuration
      thinking:
        pattern: unterminated_start
        end_marker: "</think>"
        assume_reasoning_first: true

Thinking Pattern Fields:

How It Works:

When a model has a thinking pattern configured:

1. Streaming responses are intercepted and transformed
2. Content before end_marker is sent as reasoning_content field
3. Content after end_marker is sent as content field
4. The output follows OpenAI's reasoning_content format for compatibility

Example Output:

// Reasoning content (before end marker)
{"choices": [{"delta": {"reasoning_content": "Let me analyze..."}}]}

// Regular content (after end marker)
{"choices": [{"delta": {"content": "The answer is 42."}}]}

Responses-API-only Models¶

OpenAI exposes some models exclusively via the Responses API (/v1/responses). These models are not reachable through /v1/chat/completions, so a request that targets them on a Chat Completions endpoint returns a 404 not_found from upstream.

The responses_only capability flag marks such models so the router can dispatch them to the Responses API surface instead. The flag defaults to false, so existing model entries do not need to be touched.

Configuration Example:

models:
    - id: gpt-5.4-pro
      metadata:
        display_name: "GPT-5.4 Pro"
        capabilities: ["chat", "vision", "code", "reasoning", "tool"]
        # Served only on /v1/responses; not available on /v1/chat/completions.
        responses_only: true
        limits:
          context_window: 1050000
          max_output: 128000

Models marked Responses-API-only out of the box¶

The list below is kept in sync with model-metadata.yaml and the built-in OpenAI registry (src/infrastructure/backends/openai/models/gpt5_family.rs). When a new Responses-API-only model is added upstream, both files should be updated together.

The flag follows the same lookup priority chain as the rest of the metadata (backend model_configs > model-metadata.yaml > built-in OpenAI metadata), so an operator-supplied entry can override the default for any model.

Marking a new model as Responses-API-only¶

To mark an additional model as Responses-API-only, add responses_only: true to the model entry's metadata block in any of the supported sources. Use the lookup priority that fits the deployment scope:

• model-metadata.yaml for a router-wide default that applies to every backend. Add the flag alongside the existing capability metadata; no other field needs to change. This is the recommended location for newly-released Pro models that are uniformly Responses-API-only across providers.
• Backend model_configs in config.yaml for a backend-specific override (for example, when a self-hosted clone of a Pro model is exposed on a Chat Completions endpoint and should not be dispatched to /v1/responses). A backend-level responses_only: false overrides the metadata-file default for that backend only.
• Built-in OpenAI registry in src/infrastructure/backends/openai/models/gpt5_family.rs for models that ship with the binary. New entries here should also be reflected in model-metadata.yaml so externally-loaded metadata stays consistent.

After updating any of these sources, restart the router or trigger a hot reload so the new flag takes effect on subsequent requests.

Dispatch behavior¶

The router honors responses_only=true on every public surface that would otherwise hit /v1/chat/completions:

• /v1/chat/completions: requests transparently forward to the upstream /v1/responses endpoint and the response is translated back into a strict-mode chat.completion (or chat.completion.chunk for streaming) envelope.
• /anthropic/v1/messages: the Anthropic-formatted request is converted to the Responses API shape, dispatched to /v1/responses, and the upstream response is translated back into Anthropic Messages JSON (or the Anthropic SSE event sequence for streaming). Tool-call round-trips, web-search emulation, and Unix-socket transports all branch on the flag.

In both cases the dispatch is transparent to the client: the request and response shapes match the surface the client called, so no client-side changes are required to use a responses_only model.

For Chat Completions, the bridge preserves compatible GPT-5 controls including reasoning_effort, flat verbosity, metadata, safety_identifier, prompt_cache_key, prompt_cache_options, and prompt_cache_retention. Invalid values fail explicitly instead of being silently omitted during conversion.

Backend-type constraint¶

Only OpenAI and Azure OpenAI backends serve /v1/responses. When a responses_only model is paired with a backend whose type is not OpenAI or Azure OpenAI, the router rejects the request with a 400 invalid_request_error (Anthropic-shaped on /anthropic/v1/messages, OpenAI-shaped on /v1/chat/completions) before any upstream dispatch. The message names both the model and the configured backend type so the misconfiguration is visible from the client log.

The first dispatch per (backend, model) pair logs at info level so operators can confirm Responses-API routing without enabling debug logs.

Namespace-Aware Matching¶

The router handles model IDs with namespace prefixes. For example:

• Backend returns: "custom/gpt-4", "openai/gpt-4", "optimized/gpt-4"
• Metadata defined for: "gpt-4"
• Result: All variants match and receive the same metadata

This allows different backends to use their own naming conventions while sharing common metadata definitions.

Metadata Priority and Alias Resolution¶

When looking up metadata for a model, the router uses the following priority chain:

1. Exact model ID match
2. Exact alias match
3. Date suffix normalization (automatic, zero-config)
4. Quantization / format suffix normalization (automatic, zero-config; see below)
5. Combined date + format suffix normalization
6. Wildcard pattern alias match
7. Base model name fallback (namespace stripping)

Within each source (backend config, metadata file, built-in), the same priority applies:

1. Backend-specific model_configs (highest priority)

   backends:
     - name: "my-backend"
       model_configs:
         - id: "gpt-4"
           aliases: ["gpt-4-turbo", "gpt-4-vision"]
           metadata: {...}  # This takes precedence
   

2. External metadata file (second priority)

   model_metadata_file: "model-metadata.yaml"
   

3. Built-in metadata (for OpenAI and Gemini backends)

Automatic Date Suffix Handling¶

LLM providers frequently release model versions with date suffixes. The router automatically detects and normalizes date suffixes without any configuration:

Supported date patterns:

• -YYYYMMDD (e.g., claude-opus-4-5-20251130)
• -YYYY-MM-DD (e.g., gpt-4o-2024-08-06)
• -YYMM (e.g., o1-mini-2409)
• @YYYYMMDD (e.g., model@20251130)

How it works:

Request: claude-opus-4-5-20251215
         ↓ (date suffix detected)
Lookup:  claude-opus-4-5-20251101  (existing metadata entry)
         ↓ (base names match)
Result:  Uses claude-opus-4-5-20251101 metadata

This means you only need to configure metadata once per model family, and new dated versions automatically inherit the metadata.

Automatic Quantization and Format Suffix Handling¶

Real-world model IDs arriving at /v1/models, routing logic, and backend metadata enrichment frequently combine a canonical base ID with one or more trailing quantization, format, or flavor tokens. The router strips an allowlisted set of such tokens iteratively and retries exact-id, exact-alias, and date-suffix matching after each peel, so you only need to configure metadata for the canonical base ID.

Token Categories¶

The following trailing tokens are detected and stripped (case-insensitive):

Parameter-Count Suffixes are Preserved¶

Tokens that look like parameter counts are never stripped, even when they share a trailing b:

• Kept: -32b, -70b, -8b, -4b, -a3b, -a22b, -0.6b, -1.7b, -e4b
• Stripped: -4bit, -8bit, -16bit (the literal bit suffix marks quantization)

This discrimination ensures that a parameter-count variant like qwen3-32b resolves only to explicit qwen3-32b metadata, never to a generic qwen3 entry via accidental stripping.

Layered Peeling¶

Tokens are stripped one at a time. After each peel, the router re-runs exact-id, exact-alias, and date-suffix match before attempting another peel. This lets alias configurations like gemma-3-4b-it-qat still win even when the request is gemma-3-4b-it-qat-4bit:

Request: gemma-3-4b-it-qat-4bit
         ↓ (peel -4bit)
Try:     gemma-3-4b-it-qat
         ↓ (matches alias of gemma-3-4b-qat)
Result:  Uses gemma-3-4b-qat metadata

Priority Note¶

Stripping runs after exact-id and exact-alias match. A canonical base ID that happens to end in an allowlisted token (for example gemma-3-12b-qat) wins before the peel phase runs, so existing configurations remain stable.

Suffix-Order Ambiguity¶

Both -qat-4bit and -4bit-qat orderings appear in real-world model IDs. Peeling removes one token at a time from the right, so the intermediate form mirrors the order in which tokens appeared in the input. The match sequence for gemma-3-12b-qat-4bit is gemma-3-12b-qat-4bit → gemma-3-12b-qat → gemma-3-12b, while gemma-3-12b-4bit-qat goes gemma-3-12b-4bit-qat → gemma-3-12b-4bit → gemma-3-12b. If both suffix orderings need to resolve to the same QAT-variant metadata, configure the canonical QAT base ID (gemma-3-12b-qat) with the matching metadata and let the non-QAT form (gemma-3-12b) carry its own entry; the deepest successful match wins at each peel depth. When the QAT and non-QAT variants need distinct tier or capability metadata, prefer aliases that enumerate the reorderings over relying on the peel order alone.

Length Bounds¶

The layered peel phase caps input length at 256 characters and iteration count at 8 peels as defense-in-depth against pathological inputs. Matching still runs (the exact-id and exact-alias phases remain in effect), but the peel phase short-circuits instead of walking a long allowlist-token chain. Request handlers enforce the same 256-character cap on the model field for every chat / completion / embedding endpoint, so normal traffic never hits the internal cap.

Case Insensitivity¶

Stripping is case-insensitive, so Qwen3.5-4B-4bit, QWEN3.5-4B-4BIT, and qwen3.5-4b-4bit all resolve to the same qwen3.5-4b metadata entry. Exact-id and exact-alias match phases (1 and 2) remain case-sensitive, so HuggingFace-style aliases like BAAI/bge-m3 keep their original behavior.

Wildcard Pattern Matching¶

Aliases support glob-style wildcard patterns using the * character:

• Prefix matching: claude-* matches claude-opus, claude-sonnet, etc.
• Suffix matching: *-preview matches gpt-4o-preview, o1-preview, etc.
• Infix matching: gpt-*-turbo matches gpt-4-turbo, gpt-3.5-turbo, etc.

Example configuration with wildcard patterns:

models:
  - id: "claude-opus-4-5-20251101"
    aliases:
      - "claude-opus-4-5"     # Exact match for base name
      - "claude-opus-*"       # Wildcard for any claude-opus variant
    metadata:
      display_name: "Claude Opus 4.5"
      # Automatically matches: claude-opus-4-5-20251130, claude-opus-test, etc.

  - id: "gpt-4o"
    aliases:
      - "gpt-4o-*-preview"    # Matches preview versions
      - "*-4o-turbo"          # Suffix matching
    metadata:
      display_name: "GPT-4o"

Priority note: Exact aliases are always matched before wildcard patterns. When both could match, the exact alias wins.

Using Aliases for Model Variants¶

Aliases are particularly useful for:

• Different quantizations: qwen3-32b-i1, qwen3-23b-i4 → all use qwen3 metadata
• Version variations: gpt-4-0125-preview, gpt-4-turbo → share gpt-4 metadata
• Deployment variations: llama-3-70b-instruct, llama-3-70b-chat → same base model
• Dated versions: claude-3-5-sonnet-20241022, claude-3-5-sonnet-20241201 → share metadata (automatic with date suffix handling)

Example configuration with aliases:

model_configs:
  - id: "qwen3"
    aliases:
      - "qwen3-32b-i1"     # 32B with 1-bit quantization
      - "qwen3-23b-i4"     # 23B with 4-bit quantization
      - "qwen3-16b-q8"     # 16B with 8-bit quantization
      - "qwen3-*"          # Wildcard for any other qwen3 variant
    metadata:
      display_name: "Qwen 3"
      summary: "Alibaba's Qwen model family"
      # ... rest of metadata

Aliases vs. suffix normalization: when to use which¶

Two coverage layers resolve non-canonical model ids to their owning metadata entry: explicit YAML aliases, and the layered suffix-peel allowlist in src/models/pattern_matching.rs. They are complementary, not redundant. This section explains how to choose between them when adding a new entry.

Matching phase order¶

The pipeline runs in this order, and a successful match in an earlier phase short-circuits the later ones:

1. Exact model id (case-sensitive).
2. Exact alias (case-sensitive).
3. Date-suffix normalization (-YYYYMMDD, -YYYY-MM-DD, -YYMM, @YYYYMMDD).
4. Layered quantization / format / flavor peel (case-insensitive; after each peel, exact-id + exact-alias + date-suffix phases re-run; combined date + format handled in the same loop).
5. HuggingFace repo-prefix stripping (vendor/repo -> repo) with re-entry into phases 1-4 on the stripped residual. Single-hop re-entry; phase 5 does not recurse.
6. Wildcard alias (glob-style * patterns).

A retained explicit alias runs in phase 2, strictly before the peel (phase 4) and before the prefix-strip layer (phase 5). When a retained alias and a peel-or-strip path would resolve to different metadata, the alias wins deterministically. Aliases are therefore a stronger intent signal than peel-or-strip coverage, not a weaker one.

The three alias classes¶

Every alias in model-metadata.yaml falls into one of three classes.

peel-coverable¶

Normalization reaches the same owner id without the alias, and the target metadata is the correct one. These are deletion candidates. Example: qwen3.6-35b-a3b-instruct as an alias of qwen3.6-35b-a3b. Phase 4 peels the FLAVOR token -instruct and lands on the base id directly, so the explicit alias adds no coverage. When deleting an alias in this class, add a regression assert in tests/format_suffix_normalization_test.rs::real_metadata_removed_aliases_still_resolve so the peel coverage that replaces it stays locked in.

vendor-prefix¶

The alias carries a vendor or repo prefix that suffix peel cannot strip, because peel only removes right-side tokens from a closed allowlist. Phase 5 (HuggingFace prefix stripping with re-entry into phases 1-4, where phase 4 is case-insensitive) resolves the mixed-case HF form even without the alias. Example: Qwen/Qwen3.6-35B-A3B as an alias of qwen3.6-35b-a3b. Phase 2 wins on the explicit alias, but phase 5 would also reach the base id via Qwen/ -> residual Qwen3.6-35B-A3B -> phase 4 case-insensitive match. These aliases are peel-coverable-adjacent: phase 5 makes them redundant for resolution, but they remain a deterministic intent signal. Keep them with a YAML comment noting the covering phase.

intentional-override¶

The alias deliberately routes a differently-weighted model under another entry's metadata, as an operator decision. Keep. Example: smoothie-qwen3-32b-i1 as an alias of smoothie-qwen3. The smoothie-qwen3-32b-i1 fine-tune has its own weights; the operator has chosen to surface it under the umbrella smoothie-qwen3 metadata rather than give it a dedicated entry. Peel must not infer this equivalence on its own. When an alias sits in this class, the YAML comment on the line must note that the underlying weights differ from the owner id, distinguishing an intentional override from a mechanical normalization gap.

Guidance for adding a new alias¶

Before adding a line to model-metadata.yaml, ask whether peel already covers it.

• If the new id is a canonical base with a trailing quantization, format, or flavor token already in the allowlist, and the weights are equivalent to the base metadata, do not add the alias. The peel handles it, and adding the alias would be dead code.
• If the new id shares weights with the base but ends in a token class that peel does not handle (for example, a novel fine-tune label like -abliterated or a new quantization format like -nf4), prefer extending the peel allowlist in src/models/pattern_matching.rs. This is a code change with test coverage, and it covers the entire token class in one move.
• If the new id has a vendor prefix, a repo namespace that normalization would not case-match, a parameter-count token blocking the peel chain (-Nb, -aNb, -eNb), or intentionally-different weights, add the alias with a YAML comment that states the reason. If weights differ from the owner id, say so in the comment.

Surface distinction: code-gated vs. YAML-gated¶

The peel allowlist is the strategic layer. Aliases are the tactical override and emergency channel.

Token categories already on the peel allowlist¶

The allowlist in src/models/pattern_matching.rs currently covers:

• BIT_WIDTH: -2bit, -3bit, -4bit, -5bit, -6bit, -8bit, -16bit
• GGUF_QUANT: -Q4_K_M, -Q4_K_S, -Q5_K_M, -Q6_K, -Q8_0, -Q2_K, -IQ2_XS, -IQ3_XXS, -IQ4_XS, -F16, -F32, -BF16
• FP_FORMAT: -FP4, -FP8, -FP16, -FP32, -NVFP4, -MXFP4
• INT_FORMAT: -INT2, -INT4, -INT8
• LIBRARY: -AWQ, -GPTQ, -BNB, -HQQ, -EXL2, -EXL3, -MLX
• IMATRIX: -i1 through -i8, -q2 through -q8
• UNSLOTH: -UD-Q<digit>_<KIND>, -UD-IQ<digit>_<KIND>
• CONTAINER: -GGUF, -GGML, -SAFETENSORS
• FLAVOR: -it, -instruct, -chat, -base, -thinking, -qat

Parameter-count suffixes (-Nb, -aNb, -eNb, -0.6b, -1.7b) are never peeled. They are part of canonical model identity and terminate the peel chain. This is why qwen3-32b-i1 must be kept as an explicit alias of qwen3: phase 4 strips -i1 and then halts at -32b, so without the alias the chain exhausts before reaching the base id.

HuggingFace repo-prefix stripping (phase 5)¶

Phase 5 normalizes HuggingFace-style vendor/repo prefixes off the left side of a model id, complementing the right-side suffix peel. It resolves the common HF-GGUF class where a user submits an id like unsloth/Qwen3.6-35B-A3B-GGUF and expects it to route to the canonical qwen3.6-35b-a3b metadata, without an explicit alias for every vendor x base x quant combination.

How phase 5 runs¶

1. The input is inspected for a / separator. No /, no-op.
2. Total segments (count of / plus one) must be at most MAX_PREFIX_SEGMENTS (3). org/team/repo is permitted; a/b/c/d/model is rejected outright.
3. All segments must be non-empty and free of ASCII whitespace. Malformed inputs like /repo, vendor/, vendor//repo, or vendor /repo are rejected.
4. On success, the residual is the substring after the last /. This residual is fed back into phases 1-4 with the re-entry gate closed. Phase 5 does not recurse: the inner call cannot trigger phase 5 again, so the recursion depth is exactly 1 by construction.

Composition with suffix peel¶

The re-entry runs through phase 4, so prefix stripping composes with suffix peel in a single lookup. unsloth/Qwen3.6-35B-A3B-GGUF strips to Qwen3.6-35B-A3B-GGUF, phase 4 peels -GGUF, case-insensitively matches qwen3.6-35b-a3b. This is the motivating case for the phase and covers HuggingFace GGUF forks without requiring hand-enumerated aliases.

Registered-alias precedence¶

Operators who explicitly register a vendor/repo form as a YAML alias keep deterministic control. Phase 2 runs before phase 5, so the exact alias wins before the stripping layer ever considers the input. Use this when the prefixed form must route to a different metadata entry than the canonical base id would.

Out of scope¶

• Hyphen-delimited vendor prefixes (e.g., smoothie-qwen/smoothie-qwen3-32b-i1). Different semantic class, different detection difficulty, and often represents different weights where silent base-metadata routing is the wrong call.
• Automatic vendor discovery from the HuggingFace API. The layer is purely syntactic.
• Extending the suffix peel allowlist. Orthogonal change; follow the peel-extension path for novel token classes.

Security bounds¶

Parallels the suffix peel:

Phase 5 is constant-time on adversarial input: after the segment-count, emptiness, whitespace, and length guards, the work reduces to a single slice lookup plus one additional pass through phases 1-4.

Audit procedure¶

To re-audit the YAML, run:

cargo test --test alias_audit_helper -- --ignored --nocapture audit_metadata_aliases

The helper prints every alias with its classification (REDUNDANT, LOAD-BEARING-DRIFT, LOAD-BEARING-LOSS, or WILDCARD) and the post-removal resolution target.

API Response¶

The /v1/models endpoint returns enriched model information:

{
  "object": "list",
  "data": [
    {
      "id": "gpt-4",
      "object": "model",
      "created": 1234567890,
      "owned_by": "openai",
      "backends": ["openai-proxy"],
      "metadata": {
        "display_name": "GPT-4",
        "summary": "Most capable GPT-4 model for complex tasks",
        "capabilities": ["text", "image", "function_calling"],
        "knowledge_cutoff": "2024-04",
        "pricing": {
          "input_tokens": 0.03,
          "output_tokens": 0.06
        },
        "limits": {
          "context_window": 128000,
          "max_output": 4096
        }
      }
    }
  ]
}

Request Parameter Policy¶

Use the optional request_params section to define typed defaults, operator-selected overrides, and limits for the request-parameter policy. The Router validates and hot-reloads this configuration snapshot, then applies it once at Chat Completions, Completions, Responses, and Anthropic Messages ingress before cache identity or provider routing is chosen.

request_params:
  defaults:
    max_tokens: 1024
  overrides:
    temperature: 0.0
  limits:
    max_tokens:
      min: 1
      max: 4096
  models:
    gpt-5:
      defaults:
        top_p: 0.9
      limits:
        max_tokens:
          max: 2048

Maintained examples use block-style YAML. Equivalent flow mappings such as { max_tokens: 1024 } are also accepted, and TOML configuration has the same semantics.

Evaluation Order¶

Protocol integrations apply the three modes in a fixed order:

1. defaults fills a missing or JSON/YAML-null field and preserves a value supplied by the client.
2. overrides replaces the resulting value, whether it came from the client or a default.
3. limits clamps an existing value to the inclusive min and max bounds. A limit does not create a missing field.

The top-level objects are the global policy. Each models.<name> entry may define the same three objects. Model fields override global fields independently within each mode; fields omitted from a model entry inherit the corresponding global value or range.

The runtime contract requires model names to match exactly after local alias resolution. It also requires the selected scope to be fixed before control-plane substitution, arbitrage, backend selection, retry, or fallback and remain unchanged across retries and fallbacks. Hub tier limits delivered through control_plane.policy are limits-only and intersect these local limits per bound, so neither side can broaden the other; an empty intersection rejects the request. Backend scopes, regular expressions, and glob matching are unsupported.

Supported Parameters and Validation¶

Only the following canonical parameter names are accepted:

Unknown modes or parameter names, wrong scalar types, non-finite floats, out-of-range values, and min > max are rejected with a path-qualified error. models accepts at most 64 exact scopes; each model key must be non-empty after trimming, at most 256 bytes, and free of control characters. The same parameter may appear in more than one mode because the evaluation order is deterministic. A null policy value is treated as omitted; it is not a request to force JSON null.

Runtime Provider Compatibility¶

Compatibility checks apply only to fields that the policy configured and actually changed or constrained. Client fields that the policy did not govern retain their existing provider-specific passthrough behavior. If an eligible primary or configured fallback route cannot preserve a policy-effective field, the Router rejects the request before cache lookup or upstream I/O with unsupported_request_parameter instead of silently dropping the enforced value.

Streaming fallback follows the same matrix. A cross-provider streaming hop that has no lossless native, Bedrock, Gemini, or Unix-socket translation fails explicitly; it never forwards a partly translated policy request. Each hop is rebuilt from the canonical effective request, so a multi-hop fallback cannot accumulate conversions or lose a restored field.

Response-cache lookup and storage are enabled only when routing resolves to one policy-compatible backend and no fallback chain can change that destination. The backend identity is included in the cache namespace. Requests with multiple eligible backends or an enabled fallback chain bypass exact, prefix, and streaming caches so an entry produced under one provider cannot satisfy another route.

Observability and Enforcement¶

When policy changes at least one request field and the selected provider path can preserve the effective request, the Router adds the informational x-continuum-request-params response header. Its comma-separated tokens contain only a bounded action and canonical parameter name, such as default:max_tokens or limit:temperature; request values, model names, keys, and tier identifiers are never included.

The policy is mandatory operator configuration. Clients cannot disable or bypass it with a request header, query parameter, or request-body option. The informational response header does not provide an opt-out and is not an authorization credential.

The header is present on successful responses, compatible cache hits, and upstream error responses produced after an effective mutation. It is omitted when policy is a no-op and when policy or provider compatibility rejects the transaction. A rejected transaction is rolled back and never receives a misleading applied header. Logs and Prometheus metrics follow the same bounded applied/rejected outcomes.

Hot Reload and Privacy¶

request_params is an immediate hot-reload section. A successfully parsed and validated revision atomically replaces the published configuration snapshot. A rejected revision never replaces the last-known-good snapshot. Each request captures one immutable config snapshot and one authenticated Hub tier policy before mutation, so in-flight requests, retries, and fallbacks do not change policy halfway through.

Applied-policy telemetry exposes only bounded parameter names, actions, composition source, and outcome, never request values. Config and MCP views may show the operator-authored policy, but they do not serialize prompt content or arbitrary request data and continue to redact secret-typed config fields.

Hot Reload¶

When the hot-reload feature and file watcher are active, a valid file revision is published as a new immutable configuration snapshot. Invalid revisions leave the previous snapshot active. Not every parsed field has a live runtime consumer, so use the conservative matrix below.

Applied without restart¶

• Backend additions/removals/edits (the pool is reconciled; new backends receive an immediate health check)
• Health-check intervals and thresholds
• Rate-limit and circuit-breaker policies when their services are active
• Global-prompt, request-parameter, and streaming policy read by new requests
• Fallback chain/policy edits when the fallback service already exists
• Smart-routing policy snapshots
• selection_strategy (swapped onto the live pool atomically; membership, stats, and in-flight accounting are preserved)
• Retry policy (new requests capture the updated complete policy; in-flight loops retain their entry snapshot)
• Per-request, streaming, image, model-override, and health-check timeout budgets
• All prefix_routing.* fields: enabled and max_prefix_length for request-time prefix extraction; load_factor_epsilon and virtual_nodes swapped onto the live pool (a virtual_nodes change also rebuilds the hash ring); anthropic_cache_control_injection read per-request

Restart required or currently not live¶

• server.bind_address, server.socket_mode, and server.workers
• server.connection_pool_size and CORS router layers
• Logging subscriber format/filter and tracing middleware construction
• Toggling optional services that were absent at startup, including fallback.enabled
• Response-cache backend/object construction
• timeouts.connection and timeouts.request.streaming.total (the shared HTTP-client connection/overall ceilings; request budgets still reload live)

After editing any restart-required field, validate and restart:

continuum-router config validate config.yaml

Check watcher availability and the router's capability summary through the authenticated Admin API:

curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8080/admin/config/hot-reload-status

Treat the active subsystem state, rather than the newly parsed file alone, as authoritative when a restart-required value changes.

Distributed Tracing¶

Continuum Router supports distributed tracing for request correlation across backend services. This feature helps with debugging and monitoring requests as they flow through multiple services.

Configuration¶

tracing:
  enabled: true                         # Enable/disable distributed tracing (default: true)
  w3c_trace_context: true               # Support W3C Trace Context header (default: true)
  headers:
    trace_id: "X-Trace-ID"              # Header name for trace ID (default)
    request_id: "X-Request-ID"          # Header name for request ID (default)
    correlation_id: "X-Correlation-ID"  # Header name for correlation ID (default)

How It Works¶

1. Trace ID Extraction: When a request arrives, the router extracts trace IDs from headers in the following priority order:
2. W3C traceparent header (if W3C support enabled)
3. Configured trace_id header (X-Trace-ID)
4. Configured request_id header (X-Request-ID)

5. Configured correlation_id header (X-Correlation-ID)

6. Trace ID Generation: If no trace ID is found in headers, a new UUID is generated.

7. Header Propagation: The trace ID is propagated to backend services via multiple headers:

8. X-Request-ID: For broad compatibility
9. X-Trace-ID: Primary trace identifier
10. X-Correlation-ID: For correlation tracking
11. traceparent: W3C Trace Context (if enabled)

12. tracestate: W3C Trace State (if present in original request)

13. Retry Preservation: The same trace ID is preserved across all retry attempts, making it easy to correlate multiple backend requests for a single client request.

Structured Logging¶

When tracing is enabled, all log messages include the trace_id field:

{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "info",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "message": "Processing chat completions request",
  "backend": "openai",
  "model": "gpt-4o"
}

W3C Trace Context¶

When w3c_trace_context is enabled, the router supports the W3C Trace Context standard:

• Incoming: Parses traceparent header (format: 00-{trace_id}-{span_id}-{flags})
• Outgoing: Generates new traceparent header with preserved trace ID and new span ID
• State: Forwards tracestate header if present in original request

Example traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

Disabling Tracing¶

To disable distributed tracing:

tracing:
  enabled: false

Load Balancing Strategies¶

Use the top-level selection_strategy field. The six implemented values are RoundRobin, WeightedRoundRobin, LeastLatency, Random, ConsistentHash, and PrefixAwareHash. Backend health filtering happens before selection.

selection_strategy: WeightedRoundRobin

backends:
  - name: large
    url: http://large.example.com
    weight: 3
  - name: small
    url: http://small.example.com
    weight: 1

See Load Balancing for exact semantics and current startup/hot-reload limitations.

Per-Backend Retry Configuration¶

backends:
  - name: "slow-backend"
    url: "http://slow.example.com"
    retry_override:               # Complete backend-specific retry policy
      max_attempts: 5
      initial_delay: "500ms"
      max_delay: "60s"
      backoff_multiplier: 2.0
      jitter: true
      retryable_status_codes: [429, 502, 503, 504]
      retryable_errors: [ConnectionError, TimeoutError]
      timeout: "120s"

Model Fallback¶

Configure ordered alternative models with the top-level fallback section:

fallback:
  enabled: true
  mid_stream_enabled: true
  fallback_chains:
    gpt-5.4:
      - gpt-5.4-mini
      - claude-sonnet-4-6
  fallback_policy:
    trigger_conditions:
      error_codes: [429, 500, 502, 503, 504]
      timeout: true
      connection_error: true
      model_not_found: true
      backend_unhealthy: true
    max_fallback_attempts: 3
    fallback_timeout_multiplier: 1.5
    preserve_parameters: true
  model_settings:
    gpt-5.4:
      fallback_enabled: true
      notify_on_fallback: true

The regular proxy path does not produce circuit_breaker_open; use active health checks and backend_unhealthy for inference traffic. See Model Fallback for validation, headers, metrics, cross-provider translation, and the distinction between pre-stream and mid-stream behavior.

Mid-Stream Fallback¶

Mid-stream fallback allows the router to transparently continue an active SSE stream on a fallback backend when the primary backend fails mid-response. The client's connection remains open and sees an uninterrupted response with only a brief pause during the switchover.

Mid-stream fallback activates automatically when fallback.enabled: true and a fallback chain is configured for the requested model. The streaming.mid_stream_fallback section controls how the fallback backend is invoked (continuation vs restart mode), not whether fallback happens.

Disabling Mid-Stream Buffering¶

fallback.mid_stream_enabled (default true) decouples pre-stream fallback from mid-stream buffering. When set to false while fallback.enabled: true and a chain is configured:

• The initial connection still re-routes through the fallback chain on connection errors, timeouts, or trigger error codes (pre-stream fallback is unaffected).
• Once the SSE stream starts, no stream accumulator is allocated, so there is no per-stream buffering or its associated memory cost.
• A failure that happens after the stream has started surfaces to the client as a normal stream error instead of triggering a transparent switch.

Use mid_stream_enabled: false on memory-constrained hosts, or under high concurrency with many long-context streaming sessions where the ~100-200 KB per-stream buffer adds up.

This flag is distinct from streaming.mid_stream_fallback.enabled:

fallback:
  enabled: true             # Pre-stream fallback stays on
  mid_stream_enabled: false # No per-stream buffering; mid-stream errors reach the client
  fallback_chains:
    "gpt-4o":
      - "gpt-4-turbo"
      - "gpt-3.5-turbo"

Configuration¶

fallback:
  enabled: true  # Required: enables mid-stream fallback path
  fallback_chains:
    "gpt-4o":
      - "gpt-4-turbo"
      - "gpt-3.5-turbo"

streaming:
  mid_stream_fallback:
    # Enable continuation mode (default: true).
    # When true, accumulated partial response is used to build a continuation prompt,
    # producing uninterrupted output for the client.
    # When false, the fallback backend restarts the request from scratch, which may
    # cause duplicate or incoherent content if partial output was already sent.
    enabled: true

    # Minimum estimated tokens accumulated before using continuation mode (default: 50)
    # Below this threshold the request is restarted from scratch on the fallback backend
    # instead of appending a continuation prompt.
    min_accumulated_tokens: 50

    # Maximum fallback attempts per streaming request (default: 2, max: 10)
    max_fallback_attempts: 2

    # Prompt appended as a user message after the partial assistant response
    continuation_prompt: "Continue from where you left off exactly. Do not repeat any previously generated content."

How It Works¶

1. The client sends a streaming chat completion request.
2. The router begins streaming from the primary backend, accumulating response content.

3. If the backend fails mid-stream (connection drop, timeout, error event):

   • The error is NOT forwarded to the client.
   • The accumulated partial response is captured.
   • The next healthy backend in the fallback chain is selected (unhealthy backends are skipped).
   • A continuation or restart request is sent to the fallback backend.
   • Streaming resumes on the fallback backend without closing the client connection.

4. The client receives an uninterrupted response with only a brief pause during the switchover.

Continuation vs. Restart Mode¶

The min_accumulated_tokens threshold controls which recovery mode is used:

Continuation mode (the default) produces uninterrupted output for the client. Restart mode is used automatically when there is too little context to continue meaningfully, or when the accumulated response is too long to include safely. Explicitly setting enabled: false forces restart mode unconditionally, which may cause duplicate or incoherent content visible to the client.

Edge Case Handling¶

The mid-stream fallback path addresses several edge cases automatically:

• Global timeout budget: All fallback attempts share the original request start time. Each attempt checks remaining budget before sending, preventing indefinite timeout accumulation across the chain.
• Cross-provider parameter translation: When the fallback model is on a different provider (e.g., OpenAI → Anthropic), request parameters are automatically translated: provider-specific fields removed and parameter names mapped.
• Concurrent request storms: A global semaphore (50 permits) limits simultaneous fallback attempts. Requests that cannot acquire a permit within 5 seconds are rejected gracefully.
• Accumulator truncation: When accumulated response content exceeds 100 KB, the continuation mode is forced to restart to avoid sending incoherent context to the fallback backend.
• Health re-check: Backend health is re-verified before each fallback attempt in the chain. Unhealthy backends are skipped to the next entry.
• Missing [DONE] marker: Streams ending without [DONE] but with finish_reason: "stop" are treated as completed successfully, preventing unnecessary fallback.

Metrics¶

Three Prometheus metrics track mid-stream fallback activity. See Mid-Stream Fallback Metrics for details.

Minimizing Failover Latency¶

When a backend goes down during streaming, the time until the fallback backend takes over depends on several configuration parameters across different subsystems. Below is a tuning guide for minimizing this switchover delay.

How failover delay is composed¶

The total time a client waits during a mid-stream failover is roughly:

failover_delay ≈ failure_detection_time + health_recheck_time + fallback_connection_time

Each component maps to specific configuration:

Recommended configuration for fast failover¶

# 1. Timeouts — the most impactful settings for failover speed
timeouts:
  connection: 5s               # Faster TCP connect timeout (default: 10s)
  request:
    streaming:
      first_byte: 30s          # How long to wait for the first token (default: 60s)
      chunk_interval: 10s      # Max silence between chunks before treating as failure (default: 30s)
      total: 600s              # Total streaming budget (keep generous)

# 2. Health checks — detect backend failures proactively
health_checks:
  interval: 10s                # Check every 10s instead of 30s (default: 30s)
  timeout: 3s                  # Fail health checks faster (default: 5s)
  unhealthy_threshold: 2       # Mark unhealthy after 2 failures (default: 3)
  healthy_threshold: 1         # Recover after 1 success (default: 2)
  warmup_check_interval: 1s   # Fast checks during backend startup

# 3. Fallback chain — must be configured for mid-stream fallback to activate
fallback:
  enabled: true
  fallback_chains:
    "gpt-4o":
        - "gpt-4-turbo"
        - "gpt-3.5-turbo"
  fallback_policy:
    trigger_conditions:
      error_codes: [429, 500, 502, 503, 504]
      timeout: true
      connection_error: true
      backend_unhealthy: true

# 4. Mid-stream fallback — continuation mode (default: enabled)
streaming:
  mid_stream_fallback:
    enabled: true              # Use continuation mode (default)
    max_fallback_attempts: 3   # Allow more retries for resilience (default: 2)
    min_accumulated_tokens: 30 # Lower threshold for continuation vs restart (default: 50)

Parameter impact summary¶

Failure detection scenarios¶

Different failure types are detected at different speeds:

The most common scenario in production, a backend becoming unresponsive, is governed by chunk_interval. For latency-sensitive applications, lowering this to 10–15 seconds is recommended, with model-specific overrides for slow models:

timeouts:
  request:
    streaming:
      chunk_interval: 10s      # Fast detection for most models
    model_overrides:
      gemini-2.5-pro:          # Reasoning models need longer intervals
        streaming:
          chunk_interval: 30s
          first_byte: 120s

Rate Limiting¶

The configurable, multi-dimension rate limiter is documented in Rate Limiting. Separately from it, Continuum Router has built-in, always-on protection for the model-listing endpoints, with fixed limits.

Built-in /v1/models Protection¶

GET /v1/models is rate-limited per client:

For this limiter, a client is identified by the first 16 characters of the Authorization: Bearer API key, falling back to the client IP address when no key is present. Each client has an independent quota.

POST /v1/models/refresh (force refresh) has a much tighter budget because each refresh clears the cache and fans out to every configured backend: 12 requests per minute with a burst cap of 3 per 5-second window. Only validated API keys get a per-key bucket; anonymous or invalid-token callers share a single global bucket, so spoofed headers cannot mint fresh quotas.

When a limit is exceeded, the endpoint returns 429 Too Many Requests with a message indicating whether the burst or the sustained limit was hit; rejections are also logged with the client identifier.

Cache TTL Optimization¶

To prevent cache poisoning during backend outages, empty model lists are cached for only 5 seconds initially (backing off up to 60 seconds for consecutive empty responses), while normal responses use the standard model-cache TTL.

Smart Routing¶

Smart routing classifies incoming requests by complexity and domain, then routes them to the most appropriate model tier using configurable policies. It combines a model tier registry (mapping models to tiers and domains) with a rule-based request classifier and a policy engine that maps classification results to routing decisions.

When model: "auto" is used in a chat completion request, the pipeline runs: classify the request, evaluate policies top-to-bottom, select a model from the matched tier. The same pipeline runs for all requests when intercept_all: true.

Configuration¶

smart_routing:
  enabled: true

  # Default tier when no profile matches and auto-inference is inconclusive.
  # 1 = Flagship, 2 = Standard, 3 = Lightweight. Defaults to 2.
  default_tier: 2

  # Model name that triggers smart routing. Defaults to "auto".
  virtual_model: "auto"

  # When true, all requests go through smart routing regardless of model name.
  intercept_all: false

  model_profiles:
    # Exact model name
    - model: "gpt-4o"
      tier: 1
      domains: [general, code, reasoning, creative]
      cost_per_1k_input_tokens: 0.005
      cost_per_1k_output_tokens: 0.015

    # Another exact match
    - model: "gpt-4o-mini"
      tier: 2
      domains: [general, code]
      cost_per_1k_input_tokens: 0.00015
      cost_per_1k_output_tokens: 0.0006

    # Glob pattern — matches all GGUF Q4_K_M quantized models
    - model_pattern: "*-q4_K_M"
      tier: 3
      domains: [general]

  # Routing policies: first match wins, top-to-bottom evaluation
  routing_policies:
    - name: "trivial_to_lightweight"
      when:
        complexity: [trivial, simple]
        domain: [general]
      route_to:
        tier: 3

    - name: "code_to_flagship"
      when:
        domain: [code]
        complexity: [moderate, complex, expert]
      route_to:
        tier: 1
        prefer_domains: [code]

    - name: "vision_required"
      when:
        requires: [vision]
      route_to:
        tier: 1
        require_capabilities: [vision]

    - name: "complex_to_flagship"
      when:
        complexity: [complex, expert]
      route_to:
        tier: 1

    - name: "default_to_standard"
      when: {}                      # Catch-all (always matches)
      route_to:
        tier: 2

Tier Classification¶

Domain Specialization Tags¶

Auto-Inference¶

When a model has no matching explicit profile or glob pattern, the router infers its tier automatically using three sources in priority order:

1. Pricing (from model-metadata.yaml): input cost >= $3/1k tokens maps to Flagship; >= $0.50/1k to Standard; below that to Lightweight. Zero-cost models skip pricing inference.

2. Capabilities (from model-metadata.yaml): models with 3+ high-value capabilities (vision, reasoning, audio, video, function_calling, tool) map to Flagship; 1+ such capability or 3+ total capabilities map to Standard.

3. Name heuristics: keywords like pro, ultra, opus, sonnet, turbo map to Flagship; keywords like mini, small, tiny, nano, lite, flash, haiku and quantization markers (q4_, q5_, q8_, gguf, gptq, awq) map to Lightweight.

Auto-inferred results are cached per model ID (up to 10,000 entries). The cache clears on hot-reload and when the /admin/smart-routing/model-profiles PUT endpoint is called.

Glob Pattern Syntax¶

Patterns use * as the only wildcard character. Multiple wildcards are supported.

Request Classifier¶

The rule-based classifier analyzes each request using 11 signal types and produces a ClassificationResult containing complexity level, domain tag, required capabilities, and confidence score.

Complexity Levels¶

Classification Signals¶

Each detected signal contributes to the final complexity level and domain tag. Conflicting signals (e.g., both creative and code markers present) reduce the confidence score.

LLM-Based Classifier¶

The rule-based classifier is fast but sometimes ambiguous. When that is not accurate enough, the LLM-based classifier sends the request to a small, cheap model for classification. Three operating modes are supported via classifier.method:

Hybrid mode is the recommended production setting: it adds latency only for genuinely ambiguous requests (typically 10-20% of traffic), while trivial and clear-cut requests are classified in microseconds.

smart_routing:
  enabled: true

  classifier:
    # Classification method: "rule" (default), "llm", or "hybrid".
    method: hybrid

    rule:
      # Confidence below this threshold triggers LLM escalation in hybrid mode.
      # Range: 0.0 – 1.0. Default: 0.7.
      confidence_threshold: 0.7

    llm:
      # Model used for classification. Any fast, cheap model works well.
      model: "gpt-4o-mini"

      # Backend name to route classification requests to. Must be a configured
      # backend. If omitted, the router uses the backend URL directly.
      backend: "openai-fast"

      # Maximum time allowed for a classification request (milliseconds).
      timeout_ms: 2000

      # Maximum input tokens sent to the classifier (content is truncated).
      max_input_tokens: 500

      # Number of retries after a parse failure (0 or 1). Default: 1.
      max_retries: 1

      # Temperature for classification. 0.0 gives deterministic output.
      temperature: 0.0

      # Maximum output tokens in the classification response.
      max_output_tokens: 150

      # Structured output strategy. "auto" selects the best method for the
      # configured backend: json_schema (OpenAI/vLLM), tool_use (Anthropic),
      # json_object (Ollama/Gemini/LM Studio), prompt_only (others).
      structured_output: auto   # auto | json_schema | tool_use | json_object | prompt_only

      # Include built-in few-shot examples in the system prompt.
      few_shot_examples: true

      # Custom few-shot examples appended after the built-in ones.
      custom_examples:
        - user: "What is the recommended dose of ibuprofen?"
          classification:
            complexity: simple
            domain: medical

      # Cache TTL for classification results (seconds). Default: 300.
      cache_ttl_seconds: 300

      # Maximum number of cached entries. Default: 10000.
      max_cache_entries: 10000

      # Disable the LLM classifier when load reaches this state.
      # "critical" (default) or "warning". Set to "" to never disable.
      disable_under_load_state: critical

    # Extend the built-in complexity taxonomy with custom levels.
    custom_complexity_levels:
      - name: specialized
        description: "Requires a domain-specific fine-tuned model"
        rank: 6   # Optional ordering hint (higher = harder)

    # Extend the built-in domain taxonomy with custom categories.
    custom_domains:
      - name: medical
        description: "Medical and clinical questions"

Structured Output Strategies¶

The LLM classifier needs structured JSON from the classifier model. The auto strategy picks the right mechanism based on the backend type, but you can override it explicitly:

When the classifier response cannot be parsed, the router retries once with a correction prompt (controlled by max_retries). If the retry also fails, the result from the rule-based classifier is used instead.

Classification Cache¶

Classification results are cached in memory with a configurable TTL to avoid repeated LLM calls for the same request. The cache key is a SHA-256 hash of the truncated user message, so semantically identical requests share the same cached result. The cache is per-process and not shared across router instances.

Custom Taxonomy¶

Both complexity levels and domain tags are extensible. Custom values added via custom_complexity_levels and custom_domains appear in the classifier's system prompt and are accepted in the structured-output schema. Routing policies can reference custom values just like built-in ones:

routing_policies:
  - name: specialized_to_flagship
    when:
      complexity: [specialized]
    route_to:
      tier: 1

Bypass Header¶

The LLM classifier sends an X-Smart-Route-Bypass: true header with its classification requests. The router skips smart routing for any request carrying this header, preventing circular classification loops when the classifier backend is itself behind the same router instance.

Routing Policies¶

Policies are evaluated top-to-bottom; the first match wins. If no policy matches and no catch-all is defined, the request falls back to default_tier.

Policy Condition Logic¶

• Fields within a when block are AND-ed: all specified fields must match.
• Values within a single field are OR-ed: complexity: [trivial, simple] matches either.
• when: {} is a catch-all that always matches.

Policy Fields¶

when conditions:

route_to action:

Model Selection Within a Tier¶

When multiple models belong to the matched tier, the selector scores them using:

1. Domain preference match (soft bonus)
2. Capability match (hard filter if require_capabilities is set)
3. Cost scoring (lower cost scores higher within a tier)
4. Random tiebreak for equal-score models

If no models are available in the matched tier, the router tries adjacent tiers in order (e.g., if Lightweight is empty, tries Standard, then Flagship).

Load-Aware Dynamic Tier Adjustment¶

Under normal conditions, smart routing picks the tier that best matches each request. When traffic spikes or latency rises, the load monitor tracks real-time metrics and automatically shifts routing to lighter model tiers until the system recovers.

Load management is disabled by default. Enable it under smart_routing.load_management:

smart_routing:
  enabled: true
  load_management:
    enabled: true

    # How often load metrics are evaluated (milliseconds). Default: 1000.
    assessment_interval_ms: 1000

    # Thresholds for entering Warning and Critical states.
    # Any single threshold being exceeded triggers the state transition.
    thresholds:
      warning:
        requests_per_second: 100
        avg_latency_ms: 3000
        error_rate: 0.05
        in_flight_requests: 50
      critical:
        requests_per_second: 200
        avg_latency_ms: 5000
        error_rate: 0.15
        in_flight_requests: 100

    # Routing restrictions applied per load state.
    degradation:
      warning:
        max_tier: 2           # Cap routing at Standard tier
        prefer_quantized: false
        reject_expert: false
      critical:
        max_tier: 3           # Cap routing at Lightweight tier
        prefer_quantized: true
        reject_expert: false

    # Recovery behavior.
    recovery:
      cooldown_seconds: 30    # Minimum time before downgrading the load state
      hysteresis_factor: 0.8  # Metric must drop to 80% of threshold to recover

Load States¶

When a request would normally route to Flagship (tier 1) but the load state is Warning, it is silently downgraded to Standard. The routing decision log records the adjusted policy name as <original_policy>__load_warning or <original_policy>__load_critical.

Hysteresis and Cooldown¶

Rapid oscillation between load states can itself cause instability. Two mechanisms prevent it:

• Hysteresis: to leave Warning state, a metric must drop below threshold * hysteresis_factor (default 0.8), not just below the threshold. A system that entered Warning at 100 RPS stays there until RPS drops below 80.
• Cooldown: after any state transition, recovery to a lower state is blocked for cooldown_seconds (default 30). Escalation (Normal to Warning, Warning to Critical) always bypasses the cooldown.

Per-Tier Threshold Overrides¶

If different tiers have different capacity characteristics, you can override thresholds per tier:

smart_routing:
  load_management:
    enabled: true
    thresholds:
      warning:
        requests_per_second: 100
    tier_thresholds:
      "1":                      # Tier 1 (Flagship) has a lower RPS tolerance
        warning:
          requests_per_second: 50
      "3":                      # Tier 3 (Lightweight) can handle more
        warning:
          requests_per_second: 300

Prometheus Metrics¶

When the metrics feature is enabled, the following counters and gauges are exported for load management:

The LLM classifier exports six additional metrics:

Debug Response Headers¶

Set debug_headers: true to include smart routing decision details in HTTP response headers. This is intended for development and staging environments.

smart_routing:
  enabled: true
  debug_headers: true  # Enable in development/staging

When enabled, smart-routed responses include:

Admin API¶

Smart routing exposes several admin endpoints under /admin/smart-routing/ for observability and management. The full endpoint reference is in the Admin API documentation.

Key endpoints:

• GET /status -- overall status, load state, policy count
• POST /classify -- classify a request without routing (diagnostic)
• POST /simulate -- simulate the full routing pipeline
• GET /policies and PUT /policies -- view and hot-reload policies
• GET /load-state -- current load state with assessment details
• GET /cache/stats and POST /cache/clear -- LLM classifier cache management

Structured Logging¶

All smart routing decisions are logged at DEBUG level with structured fields:

level=DEBUG msg="Smart routing decision"
  source_model="auto"
  target_model="gpt-4o-mini"
  complexity="simple"
  domain="general"
  policy="trivial_to_lightweight"
  load_state="normal"
  classifier="rule_based"
  confidence=0.92
  classification_ms=0.3

Load state transitions and policy changes are logged at INFO level.

Hot Reload¶

The smart_routing section reloads immediately when the config file changes. After a reload, the inferred-profile cache is cleared so all models are re-evaluated on the next request. Routing policies in routing_policies and load_management settings also take effect immediately without restarting the server. Policies can also be updated at runtime via the PUT /admin/smart-routing/policies endpoint.

Environment-Specific Configurations¶

Development Configuration¶

# config/development.yaml
server:
  bind_address: "127.0.0.1:8080"

backends:
  - name: "local-ollama"
    url: "http://localhost:11434"

health_checks:
  interval: "10s"                 # More frequent checks
  timeout: "5s"
  unhealthy_threshold: 3
  healthy_threshold: 2
  endpoint: "/health"

logging:
  level: "debug"                  # Verbose logging
  format: "pretty"                # Human-readable

Production Configuration¶

# config/production.yaml
server:
  bind_address: "0.0.0.0:8080"
  workers: 8                      # More workers for production
  connection_pool_size: 300       # Larger connection pool

backends:
  - name: "primary-openai"
    url: "https://api.openai.com"
    weight: 3
  - name: "secondary-azure"
    url: "https://azure-openai.example.com"
    weight: 2
  - name: "fallback-local"
    url: "http://internal-llm:11434"
    weight: 1

health_checks:
  interval: "60s"                 # Less frequent checks
  timeout: "15s"                  # Longer timeout for network latency
  unhealthy_threshold: 5          # More tolerance
  healthy_threshold: 3
  endpoint: "/health"

timeouts:
  connection: "10s"
  request:
    standard:
      first_byte: "30s"
      total: "120s"               # Bounded timeout for production
    streaming:
      first_byte: "60s"
      chunk_interval: "30s"
      total: "600s"
  health_check:
    timeout: "15s"
    interval: "60s"

retry:
  max_attempts: 3                 # Bounded retries
  initial_delay: "100ms"
  max_delay: "10s"
  backoff_multiplier: 2.0
  jitter: true
  retryable_status_codes: [429, 502, 503, 504]
  retryable_errors: [ConnectionError, TimeoutError]
  timeout: "30s"

logging:
  level: "warn"                   # Less verbose logging
  format: "json"                  # Structured logging

Container Configuration¶

# config/container.yaml - optimized for containers
server:
  bind_address: "0.0.0.0:8080"
  workers: 0                      # Auto-detect based on container limits

backends:
  - name: "backend-1"
    url: "${BACKEND_1_URL}"       # Environment variable substitution
  - name: "backend-2"
    url: "${BACKEND_2_URL}"

health_checks:
  interval: "30s"
  timeout: "5s"
  unhealthy_threshold: 3
  healthy_threshold: 2
  endpoint: "/health"

logging:
  level: "info"                   # Render a different value before validation if needed
  format: "json"                  # Always JSON in containers