Architecture Guide¶

This document covers Continuum Router's architecture, design decisions, and extension points.

Table of Contents¶

• Overview
• 4-Layer Architecture
• Library Entry Point (src/server/)
• Core Components
• Data Flow
• Dependency Injection
• Error Handling Strategy
• Extension Points
• Design Decisions
• Performance Considerations
• Rate Limiting → Configuration
• Model Fallback System → Error Handling
• Circuit Breaker → Error Handling
• File Storage → Architecture Details
• Realtime WebSocket Proxy → Configuration
• Dynamo Worker Discovery → Architecture Details
• Agent Communication Protocol (ACP)
• Backend Passthrough Contract
• Model ID Suffix Normalization → Configuration

Overview¶

Continuum Router is a high-performance, production-ready LLM API router using a 4-layer architecture that provides separation of concerns, testability, and maintainability. The architecture follows Domain-Driven Design principles and dependency inversion to create an extensible system.

Architecture Goals¶

1. Separation of Concerns: Each layer has a single, well-defined responsibility
2. Dependency Inversion: Higher layers depend on abstractions, not concrete implementations
3. Testability: Each component can be unit tested in isolation
4. Extensibility: New features can be added without modifying existing code
5. Performance: Minimal overhead while maintaining clean architecture
6. Reliability: Fail-fast design with full error handling

4-Layer Architecture¶

Layer Descriptions¶

1. HTTP Layer (src/http/)¶

Responsibility: Handle HTTP requests, responses, and web-specific concerns

Components¶

• Routes (routes.rs): Define HTTP endpoints and route handling
• Middleware (middleware/): Cross-cutting concerns (auth, logging, metrics, rate limiting)
• DTOs (dto/): Data Transfer Objects for HTTP serialization/deserialization
• Streaming (streaming/): Server-Sent Events (SSE) handling

Key Files¶

src/http/
├── mod.rs              # HTTP layer exports
├── routes.rs           # Route definitions and handlers
├── dto.rs              # Request/Response DTOs
├── handlers/           # Request handlers
│   ├── mod.rs
│   └── responses.rs    # Responses API handlers
├── middleware/         # HTTP middleware components
│   ├── mod.rs
│   ├── auth.rs         # API key authentication middleware
│   ├── admin_auth.rs   # Admin API authentication middleware
│   ├── files_auth.rs   # Files API authentication middleware
│   ├── admin_audit.rs  # Admin operations audit logging
│   ├── cors.rs         # CORS (Cross-Origin Resource Sharing) middleware
│   ├── logging.rs      # Request/response logging
│   ├── metrics.rs      # Metrics collection
│   ├── metrics_auth.rs # Metrics endpoint authentication
│   ├── model_extractor.rs # Model extraction from requests
│   ├── prometheus.rs   # Prometheus metrics integration
│   ├── rate_limit.rs   # Rate limiting middleware (legacy)
│   └── rate_limit_v2/  # Enhanced rate limiting (modular)
│       ├── mod.rs      # Module exports
│       ├── middleware.rs # Rate limiting middleware
│       ├── store.rs    # Rate limit storage and tracking
│       └── token_bucket.rs # Token bucket algorithm
└── streaming/          # SSE streaming handlers
    ├── mod.rs
    └── handler.rs      # Streaming response handling

Middleware Components¶

The HTTP layer includes several middleware components that provide cross-cutting concerns:

• auth.rs: API key authentication for main endpoints (/v1/chat/completions, /v1/models, /anthropic/*, etc.)

  • Validates API keys presented as Authorization: Bearer <key> or the Anthropic-native x-api-key: <key> header; Authorization: Bearer takes precedence when both are present
  • Supports multiple API keys configured in config.yaml
  • Returns 401 Unauthorized for invalid/missing keys

• admin_auth.rs: Separate authentication for admin endpoints (/admin/*)

  • Supports explicit none, bearer token, HTTP Basic, IP-whitelist, and shared-store API-key modes
  • Protects sensitive operations (config reload, circuit-breaker control, health management)
  • Configurable through admin.auth; a missing block fails closed and a block without method defaults to api_key

• files_auth.rs: Authentication middleware for Files API (/v1/files/*)

  • Validates API keys specifically for file upload/download/deletion operations
  • Prevents unauthorized file access and manipulation
  • Integrates with file storage service for permission checks

• admin_audit.rs: Audit logging middleware for admin operations

  • Records all admin API calls with timestamps and caller identification
  • Logs parameters and outcomes of sensitive operations
  • Provides audit trail for compliance and security monitoring
  • Configurable log levels and retention policies

• cors.rs: CORS (Cross-Origin Resource Sharing) middleware

  • Enables embedding the router in web applications, Tauri apps, and Electron apps
  • Supports wildcard origins (*), exact origins, and port wildcards (http://localhost:*)
  • Custom scheme support for desktop apps (e.g., tauri://localhost)
  • Configurable methods, headers, credentials, and preflight cache duration
  • Applied early in the middleware stack for proper preflight handling

• rate_limit_v2/: Enhanced rate limiting system (see Rate Limiting section)

  • Token bucket algorithm with per-client tracking
  • Separate limits for sustained rate and burst protection
  • Automatic cleanup of expired client entries
  • Detailed metrics for monitoring

2. Services Layer (src/services/)¶

Responsibility: Orchestrate business logic and coordinate between infrastructure components

Components¶

• Backend Service (backend_service.rs): Manage backend pool, load balancing, health checks
• Model Service (model_service.rs): Aggregate models from backends, handle caching, enrich with metadata
• Proxy Service (proxy_service.rs): Route requests, handle retries, manage streaming
• Health Service (health_service.rs): Monitor service health, track status
• Web Search Service (search/): Router-managed web_search tool for self-hosted backends. Pluggable provider trait with Serper, Exa, Brave, and SearXNG implementations, bounded tool-execution loop with iteration cap / wall-clock / byte-budget guards (see Web Search)
• System One Client (systemone/): Shared client for System One decision models across the generic, typesafe, foundry, and llama_cpp provider profiles (hosted TypeSafe Jev, Microsoft-Decision-1 on Foundry, and self-hosted Laya / Kev / Nimble / VLLM Decision 2/D3). Config block: SystemOneTargetConfig in src/core/config/models/systemone.rs. See System One Decision Models for wire details.
  • Typed queries and resilience: Executes noul, choice, score, and box questions over a state, with one bounded retry on 429 / 529 / connect failure, capped error bodies, and backend: references resolved under the pinned-backend contract.
  • Cache invalidation: cache_target_identity() includes a monotonic CacheTopologyGeneration counter that increments whenever the live Config snapshot reloads, invalidating downstream caches without hashing credentials.
  • Client caching and consumers: ClientCache (systemone/cache.rs) maintains one built client per consumer and router (or per consumer, router, and target name for /v1/decisions via get_or_build_keyed). Consumed by the guardrail provider, smart-routing classifier, action-safety evaluator, answer-quality fallback check, and public /v1/decisions native transport.
• Answer Quality (answer_quality/): Consumer-neutral answer-quality battery asking three noul questions (answered, refusal, truncated, or any subset via questions), evaluated by AnswerJudgment::worst_problem over a head-and-tail-capped {user_message, ai_response} state built from the final user message and reply. Driven by fallback.quality_check (src/proxy/quality_fallback.rs) after the guardrail output gate on non-streaming chat replies, advancing the served chain via execute_quality_hop (src/proxy/request.rs) within max_extra_hops, max_fallback_attempts, and the non-streaming chain budget.
• Usage Quality Judgment (src/control_plane/usage_quality/, control-plane feature): Extends the answer-quality battery with grounded and language_match to evaluate served chat replies (buffered or streamed) after the guardrail output gate. For route-by-route wire extraction rules, see System One Decision Models.
  • Non-blocking evaluation and call sharing: UsageQualityPlan samples the request and captures capped text before dispatch, attaching a QualityTicket to the UsageEvent. Once the reply is finalized, finish_text_with spawns the decision call in the background (bounded by a per-router in-flight cap) or reuses the judgment from fallback.quality_check when both consumers share a call.
  • Usage batch sealing: The usage loop maps, sequences, charges, and buffers records immediately, sealing each QualityTicket when forming a push batch. Judgments ready before batch formation attach to the record (deduplicated by the Hub on idempotency_key); later arrivals count as late without blocking accounting or batch pushes.
  • Streaming accumulation: Streamed requests pass a shared, take-once StreamQualityHandle on StreamObservabilityContext::usage_quality (excluding cache replays). Wire relays feed a bounded head-and-tail StreamReplyAccumulator (src/services/answer_quality/stream.rs, equivalent to cap_middle of the full reply) and settle once after the output gate's finalize: judged on a clean finish, or skipped if the gate cut/rewrote the stream, a mid-stream hop spliced generations, an error occurred, the client disconnected, or no relay claimed the handle. Every outcome carries a stream label.
  • Multi-surface ingress support: Implements QualityRequest (src/services/answer_quality/state.rs) across OpenAI Chat Completions (including the responses_only bridge in src/proxy/responses_only/bridge_usage.rs), Anthropic Messages (src/services/answer_quality/anthropic.rs, src/http/handlers/anthropic/user_turn.rs, and stream_output.rs), and /v1/responses (src/services/answer_quality/responses.rs and src/http/handlers/responses_quality.rs), using request-scoped plan slots (with_request_plan, claim_current_ticket, take_request_plan in src/control_plane/usage_quality/scope.rs) for buffered fallback chains and CompletedResponsePersister / passthrough relays for streams. Model-experiment usage records also carry experiment_id and experiment_variant_id (ExperimentArm).
• System One Classifier (smart_routing/system_one_classifier.rs): Smart-routing method: system_one / hybrid_system_one classifier.
  • Single-call scoring: Queries score (complexity levels, rounded to nearest) and choice (domains) in one decision call via the shared client, setting ClassificationResult confidence to the minimum of both answers.
  • Gating and shadow mode: SmartRouter gates results on min_confidence (falling back to rules, or escalating to LLM under hybrid_system_one), copies capabilities and language from the rule classifier, and in shadow mode runs concurrently with the active classifier to record agreement without altering routing.
  • Caching and candidate pre-filtering: Cache entries are namespaced by cache_namespace ({version, target, questions}), SystemOneClient::cache_target_identity(), and bounded post-guardrail state. Before classification and policy selection, collect_smart_routing_candidates_for_request (src/proxy/smart_routing_candidates.rs) intersects the caller API key's allowed_models and allowed_backends, backend health, circuit-breaker admission, and chat output eligibility (excluding decision-only models as excluded_incompatible).

Key Files¶

src/services/
├── mod.rs              # Module exports
├── model_service.rs    # Model aggregation service
├── proxy_service.rs    # Request proxying and routing
├── health_service.rs   # Health monitoring service
├── deduplication.rs    # Request deduplication service
├── responses/          # Responses API support
│   ├── mod.rs
│   ├── converter.rs    # Response format conversion
│   ├── router.rs       # Routing strategy determination (pass-through vs conversion)
│   ├── passthrough.rs  # Direct pass-through for OpenAI/Azure and `responses_api: native` backends
│   ├── passthrough_lifecycle.rs # GET/DELETE/cancel forwarded to the backend that served a response
│   ├── passthrough_routes.rs    # Bounded per-router (owner, response id) -> backend bindings
│   ├── session.rs      # Session management
│   ├── stream_service.rs # Streaming service orchestration
│   └── streaming.rs    # Streaming response handling
├── answer_quality/     # Answer-quality battery and decision state
│   ├── battery.rs      # QualityQuestion, AnswerJudgment, judge()
│   └── state.rs        # {user_message, ai_response[, tool_results]} state builder and caps
├── search/             # Router-managed web search service
│   ├── mod.rs          # SearchProvider trait, SearchError, SearchResult, build_provider
│   ├── executor.rs     # WebSearchExecutor: tool injection, tool-execution loop
│   └── providers.rs    # Concrete provider implementations (Serper, Exa, Brave, SearXNG)
├── streaming/          # Streaming utilities
│   ├── mod.rs
│   ├── parser.rs       # Stream parsing logic
│   └── transformer.rs  # Stream transformation (OpenAI/Anthropic)
├── tokenize/           # Engine tokenizer access, always compiled
│   ├── wire.rs         # Per-backend-type capability, paths, payloads; shared with count_tokens
│   ├── response.rs     # Engine response readers over the capped bytes, no JSON tree
│   ├── lenient.rs      # Visitors that read a wrong-typed field as absent
│   ├── token_strs.rs   # Compact token-string list: one buffer plus u32 offsets
│   ├── upstream.rs     # Time budget and capped body read, shared with count_tokens
│   ├── validate.rs     # Client contract of /tokenize and /detokenize
│   ├── dispatch.rs     # Exact-model resolution (no retry, no fallback)
│   ├── call.rs         # The upstream call, or the /apply-template then /tokenize pair
│   ├── chat_render.rs  # messages options each llama-server engine honors, and the pair's bodies
│   └── handlers.rs     # The two handlers mounted by build_api_routes
└── systemone/          # Shared System One decision-model client
    ├── mod.rs          # Module docs and re-exports
    ├── cache.rs        # ClientCache: one client per consumer, router, and target key
    ├── client.rs       # SystemOneClient: target resolution, bounded retry, decoding
    ├── metering.rs     # Attempt-level usage recording per SystemOneConsumer
    └── types.rs        # Question / Answer / Usage wire types and question validation

3. Infrastructure Layer (src/infrastructure/)¶

Responsibility: Provide concrete implementations of external systems and technical capabilities

Components¶

• Backends (backends/): Specific backend implementations (OpenAI, Anthropic, Gemini, vLLM, SGLang, Ollama, llama.cpp, MLxcel, LM Studio, Continuum Router)
• Auth (auth/): OAuth 2.0 Device Authorization Grant (RFC 8628) subsystem. The AuthStrategyRegistry is populated at startup and updated during hot-reload; every proxy hot-path handler (chat completions, responses, image generation) consults it to sign outbound requests and force-refresh on 401. The streaming request builders consult the same registry (initial dispatch, per-hop mid-stream fallback, auto backend selection, and the Unix socket transport). See src/proxy/oauth_helper.rs for the shared proxy auth-helper.
• Cache (cache/): Caching implementations (LRU/TTL-based TtlLruCache, CacheStore trait for pluggable backends, InMemoryCacheStore, SHA256-keyed ResponseCacheStore, retry-aware cache)
• KV Cache Index (kv_index/): Shared index that maps token prefix hashes to backends for KV cache-aware routing. Provides InMemoryKvIndex (DashMap + LRU eviction, single-router) and RedisKvIndex (Redis sorted sets, multi-router). Includes KvOverlapScorer which implements BackendScorer to score backends by cache overlap, load, and health. The event consumer ingests real-time cache events from vLLM SSE streams to keep the index current.
• Configuration (config/): Configuration loading, watching, validation
• HTTP Client (common/http_client.rs): HttpClientFactory presets for every backend client (the single client factory)
• Backend HTTP Protocol (backend_http/): Per-backend http_version selection (auto uses the shared proxy client; http1 and http2 use sibling clients built from ProxyClientTuning and cached process-wide). AppState::backend_client resolves the client from the routed configuration on each dispatch, and BackendClients carries that choice into mid-stream relay tasks. It also emits dispatch observability (continuum_backend_response_wait_seconds via ResponseWait and rate-limited warnings at 100 in-flight HTTP/2 requests).

Key Files¶

src/infrastructure/
├── mod.rs              # Infrastructure exports and utilities
├── auth/               # OAuth 2.0 Device Authorization Grant (RFC 8628)
│   ├── mod.rs          # Public re-exports
│   ├── device_flow.rs  # RFC 8628 state machine + ReqwestDeviceFlowClient
│   ├── token_store.rs  # Atomic write (tempfile+rename), 0600 perms, ~ expansion
│   ├── strategy.rs     # AsyncAuthStrategy trait + OAuthAuthStrategy (proactive refresh)
│   ├── registry.rs     # AuthStrategyRegistry: hot-reloadable map of backend → strategy
│   └── error.rs        # AuthError type
├── backends/           # Backend implementations
│   ├── mod.rs
│   ├── anthropic/      # Native Anthropic Claude backend
│   │   ├── mod.rs      # Backend implementation & request transformation
│   │   └── stream.rs   # SSE stream transformer (Anthropic → OpenAI)
│   ├── gemini/         # Native Google Gemini backend
│   │   ├── mod.rs      # Backend implementation with TTFB optimization
│   │   └── stream.rs   # SSE stream transformer (Gemini → OpenAI)
│   ├── openai/         # OpenAI-compatible backend
│   │   ├── mod.rs
│   │   ├── backend.rs  # OpenAI backend implementation
│   │   └── models/     # OpenAI-specific model definitions
│   ├── factory/        # Backend factory pattern
│   │   ├── mod.rs
│   │   └── backend_factory.rs  # Creates backends from config
│   ├── pool/           # Backend pooling and management
│   │   ├── mod.rs
│   │   ├── backend_pool.rs     # Connection pool management
│   │   └── backend_manager.rs  # Backend lifecycle management
│   ├── generic/        # Generic backend implementations
│   │   └── mod.rs
│   ├── llamacpp/       # llama.cpp / llama-server backend
│   │   ├── mod.rs
│   │   ├── backend.rs     # LlamaCppBackend implementation
│   │   ├── model_parser.rs # Hybrid response format parsing
│   │   └── props.rs       # /props endpoint for tool calling detection
│   ├── continuum_router/  # Continuum Router / Backend.AI GO backend
│   │   ├── mod.rs
│   │   └── backend.rs     # ContinuumRouterBackend implementation
│   └── vllm.rs         # vLLM/SGLang/Ollama/LM Studio backend implementation
├── common/             # Shared infrastructure utilities
│   ├── mod.rs
│   ├── executor.rs     # Request execution with retry/metrics
│   ├── headers.rs      # HTTP header utilities
│   ├── http_client.rs  # HTTP client factory with pooling
│   ├── statistics.rs   # Backend statistics collection
│   └── url_validator.rs # URL validation and security
├── transport/          # Transport abstraction layer
│   ├── mod.rs               # Transport enum (HTTP, UnixSocket)
│   ├── unix_socket.rs       # Unix Domain Socket client (Linux/macOS)
│   └── unix_socket_windows.rs  # Windows AF_UNIX client via socket2
├── cache/              # Caching implementations
│   ├── mod.rs
│   ├── store.rs          # CacheStore trait (async, byte-oriented cache backend abstraction)
│   ├── memory_store.rs   # InMemoryCacheStore: TtlLruCache-backed CacheStore implementation
│   ├── lru_cache.rs      # LRU cache with TTL and eviction
│   ├── response_cache.rs # SHA256-keyed LLM response cache (uses Arc<dyn CacheStore>)
│   └── retry_cache.rs    # Retry-aware cache
├── kv_index/           # KV cache prefix-hash index and overlap scoring
│   ├── mod.rs            # Re-exports KvCacheIndex, InMemoryKvIndex, RedisKvIndex, KvOverlapScorer
│   ├── index.rs          # KvCacheIndex trait + InMemoryKvIndex (DashMap/LRU) + RedisKvIndex (sorted sets)
│   ├── event_consumer.rs # KvEventConsumerManager: subscribes to vLLM SSE KV cache event streams
│   ├── scorer.rs         # KvOverlapScorer: BackendScorer impl using KV cache affinity + load + health
│   └── types.rs          # KvCacheEvent, KvCacheEventType, ConsumerStatus, KvEventConsumerConfig
├── config/             # Configuration management
│   ├── mod.rs
│   ├── loader.rs       # Configuration loading
│   ├── validator.rs    # Configuration validation
│   ├── timeout_validator.rs # Timeout configuration validation
│   ├── watcher.rs      # File watching for hot-reload
│   ├── migrator.rs     # Configuration migration orchestrator
│   ├── migration.rs    # Migration types and traits
│   ├── migrations.rs   # Specific migration implementations
│   ├── fixer.rs        # Auto-correction logic
│   ├── backup.rs       # Backup management
│   └── secrets.rs      # Secret/API key management
└── lock_optimization.rs # Lock and concurrency optimization

4. Core Layer (src/core/)¶

Responsibility: Define domain models, business rules, and fundamental abstractions

Components¶

• Models (models/): Core domain entities (Backend, Model, Request, Response)
• Traits (traits.rs): Core interfaces and contracts
• Errors (errors.rs): Domain-specific error types and handling
• Retry (retry/): Retry policies and strategies
• Scoring (scoring.rs): BackendScorer trait for composable backend scoring pipeline; each scorer assigns a signal-specific score and BackendPool sums them to select the best backend

Key Files¶

src/core/
├── mod.rs              # Core exports and utilities
├── models/             # Domain models
│   ├── mod.rs
│   ├── backend.rs      # Backend domain model
│   ├── model.rs        # LLM model representation
│   ├── request.rs      # Request models
│   └── responses.rs    # Response models (Responses API)
├── traits.rs           # Core traits and interfaces
├── errors.rs           # Error types and handling
├── container.rs        # Unwired legacy (not used at runtime)
├── scoring.rs          # BackendScorer trait: composable async scoring pipeline for backend selection
├── hashing.rs          # SHA256 prefix key computation (compute_hash)
├── prefix_key.rs       # Prefix key extraction from request bodies
├── async_utils.rs      # Async utility functions
├── duration_utils.rs   # Duration parsing utilities
├── streaming/          # Streaming models
│   ├── mod.rs
│   └── models.rs       # Streaming-specific models
├── retry/              # Retry mechanisms
│   ├── mod.rs
│   ├── policy.rs       # Retry policies
│   └── strategy.rs     # Retry strategies
├── circuit_breaker/    # Circuit breaker pattern
│   ├── mod.rs          # Module exports
│   ├── config.rs       # Configuration models
│   ├── state.rs        # State machine and breaker logic
│   ├── error.rs        # Circuit breaker errors
│   ├── metrics.rs      # Prometheus metrics
│   └── tests.rs        # Unit tests
├── files/              # File processing utilities
│   ├── mod.rs          # Module exports
│   ├── resolver.rs     # File reference resolution in chat requests
│   ├── transformer.rs  # Message transformation with file content
│   └── transformer_utils.rs # Transformation utility functions
├── tool_calling/       # Tool calling transformation
│   ├── mod.rs          # Module exports
│   ├── definitions.rs  # Tool/function type definitions
│   └── transform.rs    # Backend-specific transformations
└── config/             # Configuration models
    ├── mod.rs
    ├── models/            # Configuration data models (modular structure)
    │   ├── mod.rs         # Public module exports
    │   ├── config.rs      # Main Config struct, ServerConfig, BackendConfig
    │   ├── backend_type.rs # BackendType enum definitions
    │   ├── model_metadata.rs # ModelMetadata, PricingInfo, CapabilityInfo
    │   ├── global_prompts.rs # GlobalPrompts configuration
    │   ├── samples.rs     # Sample generation configurations
    │   ├── validation.rs  # Configuration validation logic
    │   └── error.rs       # Configuration-specific errors
    ├── timeout_models.rs # Timeout configuration models
    ├── cached_timeout.rs # Cached timeout resolution
    ├── optimized_retry.rs # Optimized retry configuration
    ├── metrics.rs      # Metrics configuration
    └── rate_limit.rs   # Rate limit configuration

Library Entry Point (src/server/)¶

Responsibility: Expose Continuum Router as an embeddable Rust library crate via a ContinuumRouter builder API

The src/server/ module decouples all initialization logic from the binary entry point (main.rs), enabling downstream Rust projects to embed the router directly without spawning a subprocess. The binary main.rs becomes a thin CLI wrapper (~220 lines) that delegates to this module.

Key Files:

src/server/
├── mod.rs      # ContinuumRouter struct, ContinuumRouterBuilder with fluent API
├── init.rs     # Individual service initialization functions (HTTP client, health checker,
│               # circuit breaker, file service, API key store, rate limiting)
├── state.rs    # initialize_services() - assembles all services into AppState
├── routes.rs   # Route builders for API, admin, files, WebUI, metrics, and CORS
└── serve.rs    # Server lifecycle: bind listeners, hot-reload, graceful shutdown

Builder API:

use continuum_router::{ContinuumRouter, Config};

// From a config file (enables hot-reload when called)
let router = ContinuumRouter::from_config_file("config.yaml")
    .await?
    .enable_hot_reload(true)
    .build()
    .await?;
router.serve("0.0.0.0:8080").await?;

// From a programmatic Config struct (no file dependency)
let router = ContinuumRouter::from_config(Config::default())
    .enable_health_checks(false)
    .with_config_dir("/etc/continuum")
    .build()
    .await?;

// Embed into an existing Axum application
let app = axum::Router::new()
    .nest("/llm", router.into_router());

Builder Override Semantics:

Each enable_* method registers an explicit override. When None (not called), the builder respects whatever the config file or Config::default() says. When Some(value), the override wins regardless of the config source.

ContinuumRouter struct (returned by build()):

Design Decisions:

• initialize_prompt_file_cache accepts Option<&Path> instead of &Args, removing the CLI dependency from library code.
• perform_container_health_check (Docker HEALTHCHECK) remains in main.rs because it is binary-only and creates no TCP listening sockets.
• load_configuration (CLI fallback logic) also remains in main.rs to keep the library free of CLI concerns.
• All init::* and state::* functions are pub(crate) or pub(super); only ContinuumRouter, ContinuumRouterBuilder, and the init / routes sub-modules are re-exported from lib.rs.

Core Components¶

Backend Pool¶

Location: src/infrastructure/backends/pool/ (backend_pool.rs + lifecycle.rs)

Purpose: The single source of truth for backend membership, selection, and lifecycle. It holds Arc<dyn Backend> execution objects, runs the health-aware selection (round-robin / weighted / least-latency / consistent-hash / prefix-aware, plus the composable scorer pipeline with in-flight counting and per-backend stats), and owns the graceful-drain and hot-reload lifecycle.

Every live HTTP request path (chat completions, streaming chat, Responses, Anthropic Messages and count_tokens, image generation) dispatches its final backend pick through this pool:

• Candidate resolution and selection: Each path resolves and pre-filters its own candidate set (model lookup, internal-backend filter, per-key allow-list, retry-state exclusion) and hands the names to the shared selection seam (proxy::selection::select_pooled_backend). The seam health-filters the candidates and calls BackendPool::select_from_candidates so the configured selection_strategy and any registered scorers (such as KvOverlapScorer when prefix_routing.enabled and a KV index are configured) govern traffic distribution instead of candidate-list order.
• Consistent-hash ring cache: The consistent-hash ring cache is a small bounded LRU keyed by the participating backend-name vector (including order), so alternating per-model candidate subsets reuse their own rings without unbounded growth. Each selection call clones an Arc ring snapshot before walking it, preventing a ring built for one candidate set from being interpreted against another and keeping the cache lock out of the ring walk.
• Compile-time visibility enforcement (UserRoutableCandidates): Candidate names travel as proxy::selection::UserRoutableCandidates rather than a bare Vec<String>. Because the user-routing visibility filter must run before the per-key allow-list (so a model served only by internal: true or enabled: false backends reports model-not-found rather than a misleading 403 Forbidden), UserRoutableCandidates encodes that proof at the type level:
  • Its only ordinary constructors are filter_user_routing_candidates (names) and filter_user_routing_backends (typed BackendConfig entries). All other operations are narrowings that cannot admit a hidden backend, so any dispatch path that omits the filter fails to compile.
  • UserRoutableCandidates::bypass_for_management is the sole explicit bypass and has no callers; router-internal consumers that legitimately address hidden backends (management probes, health monitoring, the admin catalog, hub inventory, model discovery) resolve backends without going through the user-routing seam.

pub struct BackendPool {
    backends: Arc<RwLock<Vec<Arc<dyn Backend>>>>,
    counter: Arc<AtomicUsize>,
    selection_strategy: SelectionStrategy,
    stats: Arc<RwLock<HashMap<String, BackendStats>>>,
    scorers: Arc<RwLock<Vec<Arc<dyn BackendScorer>>>>,
    // Lifecycle side tables (keyed by backend name), see lifecycle.rs:
    backend_states: Arc<RwLock<HashMap<String, BackendState>>>, // Active/Draining/Removed
    draining: Arc<RwLock<Vec<DrainingEntry>>>,                  // + drain timestamps
    // ... hash-ring cache, epsilon, scorer threshold ...
}

impl BackendPool {
    // Health/scorer-aware selection over the live backends.
    pub async fn select_backend(&self, model: Option<&str>) -> CoreResult<Arc<dyn Backend>> { /* ... */ }

    // Candidate-restricted selection used by every live HTTP path:
    // intersects the caller's pre-filtered candidate names with pool membership
    // (by name only, preserving order), then runs the scorer pipeline and the
    // configured strategy over exactly that subset.
    pub async fn select_from_candidates(
        &self,
        candidates: &[String],
        model: Option<&str>,
        context: &ScoringContext,
    ) -> CoreResult<Arc<dyn Backend>> { /* ... */ }

    // Graceful drain (out of rotation, in-flight requests finish) + cleanup.
    pub async fn drain_backend(&self, name: &str) -> Option<Arc<dyn Backend>> { /* ... */ }
    pub async fn cleanup_draining(&self) -> usize { /* ... */ }

    // In-place hot reload: add/drain/recreate backends from new config.
    pub async fn update_from_config(&self, configs: &[BackendConfig], factory: &BackendFactory) { /* ... */ }
}

The mutable lifecycle state lives in side tables keyed by backend name (backend_states, draining), mirroring the pool's existing name-keyed stats map, so the hot selection path keeps its plain Vec<Arc<dyn Backend>> layout. AppState holds exactly one backend_pool; the proxy/passthrough layers project the chosen Arc<dyn Backend> to a lightweight PooledBackend { name, url } handle at the selection boundary.

The pool also owns the per-backend engine statistics store (src/infrastructure/backends/engine_stats/), whose background poller reads each self-hosted engine's load or metrics endpoint (vLLM /metrics, SGLang /v1/loads, llama.cpp /slots and /metrics, Ollama /api/ps, LM Studio /api/v0/models) and normalizes it into a TTL-bounded EngineStatsSnapshot. Prometheus scrape refresh (backend_engine_*{backend}), GET /admin/backends/{name}/engine-stats, WebUI dashboard cards, Backend::get_current_load(), the KV overlap scorer, and selection saturation admission hints (routing.engine_load) all read this store (see Metrics).

Health Checker¶

Location: src/health.rs → src/services/health_service.rs

Purpose: Monitor backend health with configurable thresholds, automatic recovery, and accelerated warmup detection

pub struct HealthChecker {
    backends: Arc<RwLock<Vec<Backend>>>,
    config: HealthConfig,
    status_map: Arc<RwLock<HashMap<String, HealthStatus>>>,
}

pub struct HealthConfig {
    pub interval: Duration,
    pub timeout: Duration,
    pub unhealthy_threshold: u32,     // Failures before marking unhealthy
    pub healthy_threshold: u32,       // Successes before marking healthy
    pub warmup_check_interval: Duration, // Accelerated interval during warmup (default: 1s)
    pub max_warmup_duration: Duration,   // Max time in warmup mode (default: 300s)
}

pub enum HealthStatus {
    Healthy,      // Backend responding with HTTP 200
    Unhealthy,    // Connection failure or error
    WarmingUp,    // HTTP 503 - backend loading (accelerated checks)
    Unknown,      // Initial state
}

Accelerated Warmup Health Checks¶

When a backend returns HTTP 503 (Service Unavailable), it enters the WarmingUp state. During this state:

• Health checks run at warmup_check_interval (default: 1 second) instead of the normal interval
• This reduces model availability detection from ~30 seconds to ~1 second
• After max_warmup_duration, the backend is marked as Unhealthy
• Particularly useful for llama.cpp backends that return HTTP 503 during model loading

Transport Layer¶

Location: src/infrastructure/transport/

Purpose: Provide a unified transport abstraction for backend communication over HTTP/HTTPS or Unix Domain Sockets

The transport layer enables secure local LLM communication via Unix sockets, eliminating the need for TCP port exposure.

URL Schemes¶

Transport Enum¶

pub enum Transport {
    Http { url: String },
    UnixSocket { socket_path: PathBuf },
}

impl Transport {
    pub fn from_url(url: &str) -> Result<Self, TransportError>;
    pub fn is_unix_socket(&self) -> bool;
    pub fn is_http(&self) -> bool;
}

Unix Socket Client¶

The UnixSocketClient provides HTTP-over-Unix-socket communication:

pub struct UnixSocketClient {
    socket_path: PathBuf,
    config: UnixSocketClientConfig,
}

impl UnixSocketClient {
    pub async fn get(&self, endpoint: &str, headers: Option<Vec<(String, String)>>)
        -> Result<UnixSocketResponse, UnixSocketError>;
    pub async fn post(&self, endpoint: &str, headers: Option<Vec<(String, String)>>, body: Bytes)
        -> Result<UnixSocketResponse, UnixSocketError>;
    pub async fn health_check(&self) -> Result<bool, UnixSocketError>;
}

Security Features¶

• Path Traversal Protection: Validates socket paths to prevent directory traversal attacks
• CRLF Injection Protection: Validates endpoints and headers for HTTP header injection
• Response Size Limits: Configurable max response size (default: 100MB)

Platform Support¶

Windows AF_UNIX support uses the socket2 crate for socket creation and raw Winsock handle conversion to tokio-compatible types. The implementation is isolated in unix_socket_windows.rs and activated via #[cfg(windows)] gates.

Model Aggregation Service¶

Location: src/models/ (modular structure)

Purpose: Aggregate and cache model information from all backends, enrich with metadata, with cache stampede prevention

Module Structure:

src/models/
├── mod.rs             # Public module exports
├── types.rs           # Model, AggregatedModel, ModelList, SingleModelResponse types
├── metrics.rs         # ModelMetrics tracking (includes stampede metrics)
├── cache.rs           # ModelCache with stale-while-revalidate & singleflight
├── config.rs          # ModelAggregationConfig
├── fetcher.rs         # Model fetching from backends
├── handlers.rs        # HTTP handlers for /v1/models and /v1/models/{model} endpoints
├── background_refresh.rs  # Background periodic cache refresh service
├── pattern_matching.rs    # 6-phase model ID lookup pipeline (date / format suffix normalization, HuggingFace repo-prefix stripping, wildcard)
├── utils.rs           # Utility functions (normalize_model_id, etc.)
└── aggregation/       # Core aggregation logic
    ├── mod.rs         # ModelAggregationService implementation
    └── tests.rs       # Unit tests

For the model ID resolution pipeline used during metadata enrichment, see Automatic Quantization and Format Suffix Handling in the configuration guide. For policy guidance on when a new mapping should be added as an explicit YAML alias versus an extension to the peel allowlist or the HuggingFace prefix-strip layer, see Aliases vs. suffix normalization: when to use which. The prefix-strip layer composes with the suffix peel: a single lookup can strip a vendor/repo prefix and then peel an allowlisted suffix token, so unsloth/Qwen3.6-35B-A3B-GGUF routes to qwen3.6-35b-a3b metadata without a hand-registered alias.

Cache Stampede Prevention¶

The model aggregation service implements four strategies to prevent cache stampede (thundering herd problem):

1. Singleflight Pattern: Only one aggregation request runs at a time. Concurrent requests wait for the ongoing aggregation to complete, then share the result.

2. Stale-While-Revalidate: When cache is stale (between soft and hard TTL), return stale data immediately while triggering a background refresh. Clients get fast responses with potentially slightly stale data.

3. Background Periodic Refresh: A background task proactively refreshes the cache before expiration. Requests never block on cache refresh (except cold start).

4. Supersession Stamps: Every aggregation records the pool's membership_generation and the cache's invalidation_epoch immediately before snapshotting the pool, and compares both at store time:

   • Immediate expiration on stamp drift: If either stamp moved during the fan-out (a backend added, removed, or drained by hot reload, the Admin API, or AppProxy, or a clear_cache() call from a configuration change that left membership identical), the result is stored as an already-expired entry instead of Fresh. Readers within the stale-serve bound still receive the previous list while a follow-up refresh starts immediately (in-place bounded at three passes for background revalidation, or as one spawned background task on the request path).
   • Race-free ordering: Stamps are read before taking the pool snapshot. Because the membership comparison is not fused with the insert lock like the epoch comparison, the store re-reads membership_generation after insertion and expires the entry in place if it moved. Hot reload bumps both stamps back-to-back before its health-check round trip, so a single reload reaches an in-flight fan-out as one event.
   • Bounded in-place retry: Between passes, the in-place retry releases the aggregation lock and backs off with jitter so a POST /v1/models/refresh queued behind a pass does not wait for the entire chain. If every pass in a chain is superseded, the entry remains expired with a 2 s re-arm hold before the next revalidation (lifted immediately by any Fresh publish).

pub struct ModelAggregationService {
    cache: ModelCache,  // With singleflight lock and background refresh tracking
    config: ModelAggregationConfig,
    fetcher: ModelFetcher,
}

impl ModelAggregationService {
    // Aggregate models with singleflight protection
    pub async fn aggregate_models_with_singleflight(&self, state: &Arc<AppState>)
        -> Result<AggregatedModelsResponse, StatusCode> { /* ... */ }

    // Find backends with stale-while-revalidate support
    pub async fn find_backends_for_model(&self, state: &Arc<AppState>, model_id: &str)
        -> Vec<String> { /* ... */ }

    // Clear cache (used during hot reload)
    pub fn clear_cache(&self) { /* ... */ }
}

Cache TTL Configuration¶

The cache uses a dual-TTL approach:

Metrics¶

New metrics for cache stampede monitoring:

• stale_while_revalidate: Requests that returned stale data during refresh
• coalesced_requests: Requests that waited for ongoing aggregation
• background_refreshes: Background refresh operations initiated
• background_refresh_successes/failures: Background refresh outcomes
• singleflight_lock_acquired: Times the aggregation lock was acquired

Proxy Module¶

Location: src/proxy/ (modular structure)

Purpose: Handle request proxying, backend selection, file resolution, and image generation/editing

Module Structure:

src/proxy/
├── mod.rs             # Public module exports
├── backend.rs         # Backend selection and routing logic
├── request.rs         # Request execution with retry logic
├── files.rs           # File reference resolution in requests
├── image_gen.rs       # Image generation handling (DALL-E, Gemini, GPT Image)
├── image_edit.rs      # Image editing support (/v1/images/edits)
├── image_utils.rs     # Image processing utilities (multipart, validation)
├── handlers.rs        # HTTP handlers for proxy endpoints
├── utils.rs           # Utility functions (error responses, etc.)
└── tests.rs           # Unit tests

Key Responsibilities¶

• Backend Selection: Intelligent routing to available backends
• File Resolution: Resolve file references in chat requests
• Image Generation: Support for OpenAI (DALL-E, GPT Image) and Gemini (Nano Banana) image models
• Image Editing: Image editing and variations endpoints
• Request Retry: Automatic retry with exponential backoff
• Error Handling: Standardized error responses in OpenAI format
• Internal Streaming: Stream eligible buffered chat and completions requests from the backend so a client cancel is metered (proxy.internal_streaming)

Internal streaming of buffered requests¶

proxy.internal_streaming (src/proxy/internal_streaming.rs and internal_streaming_read.rs) transforms eligible buffered requests on the wire inside make_http_request: after all payload transforms run, the dispatch is sent with stream: true and stream_options shaped by the shared apply_router_stream_options writer (include_usage: true, plus continuous_usage_stats: true on vllm backends; any client stream_options is discarded first).

• Scope and route eligibility: The chat completions and completions handlers run their post-policy work inside a task-local scope (client_route), so their retries, fallback hops, deduplication, web-search turns, quality hops, and disaggregated or shim dispatch qualify. The /v1/decisions emulation hop does not qualify, and the Anthropic Messages and Responses ingresses, embeddings, rerank, images, files, synthetic probes (which use Backend trait executors), and router-initiated calls never reach this arm.
• Backend and request conditions: Within client_route, eligibility requires the backend's effective internal-streaming flag, an OpenAI-wire backend type (excluding azure), HTTP(S) transport, and a compatible request shape (n: 1, no logprobs, response_format of text or json_object, and none of functions, function_call, audio, audio modalities, or web_search_options, whose output the assembler does not rebuild).

A non-2xx reply is read and classified, and a 2xx application/json reply is passed through unchanged. A 2xx stream is read by one relay loop under the dispatch's own timeout, parsed with SseParser in standard mode, fed to a StreamUsageFinalizer, and assembled by services::streaming::assemble (shared with streaming cache replay) into a standard buffered response body. The read settles in a single tail (enforced by tests/stream_early_exit_audit_test.rs):

• Failure before any output: If no data event arrived, an error event arrived first, a transport error or timeout occurred after a role-only chunk, or the body ended with neither [DONE] nor a finish_reason, nothing is recorded by the internal-streaming reader and the failure retries like a normal buffered body read (covered by the retry seam's per-attempt record). Output is defined as a non-zero router delta count, a reported engine completion count, or any chat delta field beside role with a non-empty value (so a refusal still makes the failure final).
• Completed stream: Leaves its finalized usage on the request ledger keyed by the attempt. The row recording that attempt (the served row, a superseded web-search turn, or a quality hop) consumes it as a buffered row: token counts, outcome (usage_missing without engine usage), and token source come from the stream, while TTFT remains the response-header observation and stream stays false.
• Failure after output: Including a body that ends without [DONE] and without any finish_reason, returns a BackendRequestFailure carrying the finalized partial usage (partial). The retry seam records the attempt (upstream_error or timeout_mid_stream, with the partial count) and treats it as final like replay: disabled (TriggerReason::PartialOutput), preventing retries, fallback hops, or disaggregated unified fallback from re-sending the request.
• Client disconnect: A read dropped after the first event (because the client disconnected and the handler future was dropped) synchronously records one client_cancelled attempt from Drop (when inside a Tokio runtime) and drops the upstream body unread. The circuit breaker records settle_success on an assembled body and settle_error on a failure.

Retry Handler¶

Location: src/services/deduplication.rs

Purpose: Implement exponential backoff with jitter and request deduplication

EnhancedRetryHandler composes the core retry engine with a DeduplicationManager; configuration uses the public string-duration RetryConfig (max_attempts, initial_delay, max_delay, backoff_multiplier, jitter, retryable status/error lists, and total timeout). The retry state machine itself lives in src/core/retry/strategy.rs.

Per-Backend Replay Policy and Attempt Tracking¶

• Replay policy (backends[].replay): Enforced at execute_single_retry_attempt_observed (src/proxy/request.rs), the single seam every retry attempt passes through. A failed dispatch to a replay: disabled backend that is not provably undelivered (BackendRequestFailure::not_sent, set only on a failed connect or a router-side refusal before sending) terminates the retry loop and raises a per-request barrier that sets the fallback chain's trigger to TriggerReason::ReplayDisabled (which never hops). The streaming connection phase and mid-stream relay, provider-shaped ingress chains (src/http/ingress_fallback.rs::seal_replay_barrier), the quality-hop driver, and disaggregated unified fallback apply the same rule. Requests that a replay: disabled backend may serve also skip deduplication. (RetryState::backend_attempted is deprecated and unused by selection.)
• Attempt tracking: The same seam begins an attempt on the request ledger (begin_attempt in src/services/stats/request_ledger_attempts.rs) after selection, the hop dial bound, and circuit admission succeed, immediately before the request leaves, so a refusal prior to dispatch is never recorded as an attempt.
  • Each attempt claims the next attempt_seq of its client request and names the previous attempt as parent_attempt.
  • On error, the seam immediately records the failed attempt via record_attempt_usage (src/services/stats/attempts.rs) before any retry sleep or chain hop, capturing BackendRequestFailure::upstream_status and any token usage reported in the error body.
  • The terminal attempt is recorded by the request-level recorder, which stamps the ledger's current attempt (a request-level failure row records no attempt). The pre-stream dials of connect_with_pre_stream_fallback, the Anthropic web-search invoker, and the web-search loop's superseded-turn hook use the same two calls.

Request Deduplication (In-Flight Coalescing)¶

Non-streaming POST requests under 10 KB that no replay: disabled backend may serve and that pass may_coalesce (src/services/deduplication/eligibility.rs) are coalesced in flight by execute_with_deduplication_attributed:

• Endpoint and temperature eligibility:
  • /v1/chat/completions and /v1/completions are admitted only with an explicit, numerically zero temperature (an omitted temperature is not zero because OpenAI defaults to 1, whereas the response cache treats a missing value as 0). Eligibility inspects the request body before dispatch, so a backends[].request_extensions.body.overrides.temperature applied inside the attempt is not visible.
  • /v1/embeddings, /v1/rerank, /embed_sparse, and /v1/models are always eligible, as is the /v1/decisions emulation hop (which sends temperature: 0).
  • /v1/images/generations and all other endpoints are never eligible. The web-search tool loop and answer-quality fallback run their turns and hops inside without_coalescing so each turn or hop is recorded as its own attempt with its own tokens.
• Leader election and in-flight waiter lifecycle:
  • The first request admitted under a key becomes the leader and owns an in-flight marker in the DashMap via a single atomic DashMap::entry operation (DeduplicationManager::admit), guaranteeing two identical requests never both lead.
  • Subsequent identical requests join while the marker is in flight and wait on its tokio::sync::watch channel (src/services/deduplication/waiters.rs) for the leader's outcome (success or error, so waiters see the leader's 503 or 429 with Retry-After rather than a 408).
  • When the leader settles, it removes its marker before publishing the outcome to joined waiters; publishing first would hand a settled result to a new arrival between publish and removal. Reusing completed answers is handled by the response cache.
  • The wait budget is computed from the leader's maximum allowed execution time: one attempt's total timeout (standard.total or image-generation total, scaled for fallback hops and clamped by timeouts.limits), plus retry.timeout (clamped by max_retry_timeout), plus timeouts.connection on a fallback hop (for the hop dial permit). Expiry fails the waiter with 408 timeout.
  • If a leader disconnects before finishing, its drop guard removes the marker and closes the channel; waiting requests re-join once with one promoted to leader, and a waiter that loses a second leader executes its own retry loop without deduplication.
• Waiter accounting (src/services/stats/coalesced.rs):
  • A waiter receiving the leader's result is accounted like a local response-cache replay (note_coalesced_result): recorded under no backend (not_dispatched) and attempt_seq: 0 (even if an earlier attempt failed; Hub mapping floors the sequence at 1 and drops any parent), with the leader's prompt tokens and zero completion tokens.
  • It emits a usage export record with cache_hit_type: exact and, on control-plane builds, a metadata-only record_local_cache_hit event that charges no token window or budget. Request-count rate limiting in middleware still counts every caller.
• Deduplication key composition (src/services/deduplication/key_headers.rs):
  • Keys combine the configuration snapshot, endpoint, dispatched body (after smart routing, Hub substitution, and alias or experiment dispatch rewrite model), effective backend allow-list, and a closed allow-list of inbound headers matched case-insensitively in canonical order:
    • Credential headers (is_credential_header: authorization, x-api-key, x-goog-api-key, etc.).
    • Provider headers affecting upstream behavior (anthropic-version, anthropic-beta, openai-organization, openai-project, openai-beta, x-vllm-priority).
    • Router control headers (x-backend, x-continuum-router-hops, x-smart-route-bypass, and Hub opt-outs x-continuum-substitution, -arbitrage, -budget-fallback, -compression, -batch).
    • Dynamo shim worker-picker pins (x-dynamo-prefill-instance-id, x-dynamo-prefill-dp-rank, x-dynamo-worker-instance-id, x-dynamo-dp-rank; client-supplied x-dynamo-* headers are stripped before the handler).
    • Headers referenced by the active configuration snapshot (smart_routing.routing_policies[].when.header and model_experiments[].sticky_key: header:<name>).
  • All other headers (X-Trace-ID, traceparent, x-request-id, user-agent, x-stainless-*) are excluded. Because callers sharing an API key (or unauthenticated callers) share one leader generation, callers that must not share results should use separate API keys.

pub struct EnhancedRetryHandler {
    pub retry_handler: RetryHandler,
    deduplication: DeduplicationManager,
    enable_deduplication: bool,
}

Circuit Breaker¶

Location: src/core/circuit_breaker/

Purpose: Maintain an independent per-backend three-state circuit-breaker state machine

pub struct CircuitBreaker {
    states: Arc<DashMap<String, BackendCircuitState>>,
    config: CircuitBreakerConfig,
    metrics: Option<CircuitBreakerMetrics>,
}

pub struct CircuitBreakerConfig {
    pub enabled: bool,
    pub failure_threshold: u32,           // Failures before opening (default: 5)
    pub failure_rate_threshold: f64,      // Failure rate threshold (default: 0.5)
    pub minimum_requests: u32,            // Min requests before rate calculation
    pub timeout: Duration,                // Open-state cooldown (default: 60s)
    pub half_open_max_requests: u32,      // Max requests in half-open state
    pub half_open_success_threshold: u32, // Successes needed to close
}

pub enum CircuitState {
    Closed,    // Normal operation - requests pass through
    Open,      // Failing fast - requests rejected immediately
    HalfOpen,  // Testing recovery - limited requests allowed
}

Key Features¶

• Per-backend circuit breakers with independent state
• Atomic operations for lock-free state checking in hot path
• Automatic state transitions based on success/failure patterns
• Sliding window for failure rate calculation
• A metrics collector that can be attached by an embedding application
• Admin endpoints for inspection and manual control

The standard LLM proxy path records backend outcomes and consults the breaker during candidate selection and immediately before dispatch. Open circuits are removed from ordinary inference traffic, half-open probes are capacity-limited, and the normal metrics registry exposes the live breaker counters. Active health filtering, retry, and fallback remain complementary layers.

Container (Unwired Legacy)¶

Location: src/core/container.rs

Container (src/core/container.rs) is a leftover from an earlier design that was never wired up at runtime. It is constructed only inside its own unit tests and registers no service.

The router does not use a runtime DI container. Services are ordinary constructor arguments; see Dependency Injection.

Data Flow¶

Request Processing Flow¶

sequenceDiagram
    participant Client
    participant HTTPLayer as HTTP Layer
    participant ProxyService as Proxy Service
    participant BackendService as Backend Service
    participant ModelService as Model Service
    participant Backend as LLM Backend

    Client->>HTTPLayer: POST /v1/chat/completions
    HTTPLayer->>HTTPLayer: Apply Middleware (auth, logging, metrics)
    HTTPLayer->>ProxyService: Forward Request

    ProxyService->>ModelService: Get Model Info
    ModelService->>ModelService: Check Cache
    alt Cache Miss
        ModelService->>BackendService: Get Backends for Model
        BackendService->>Backend: Query Models
        Backend-->>BackendService: Model List
        BackendService-->>ModelService: Filtered Backends
        ModelService->>ModelService: Update Cache
    end
    ModelService-->>ProxyService: Model Available on Backends

    ProxyService->>BackendService: Select Healthy Backend
    BackendService->>BackendService: Apply Load Balancing
    BackendService-->>ProxyService: Selected Backend

    ProxyService->>Backend: Forward Request
    Backend-->>ProxyService: Response (streaming or non-streaming)

    ProxyService->>ProxyService: Apply Response Processing
    ProxyService-->>HTTPLayer: Processed Response
    HTTPLayer-->>Client: HTTP Response

Model Name Resolution and Model Experiments¶

Before backend selection, every model-routed ingress resolves the name it dispatches through one seam, dispatch_model_for_request in src/proxy/alias_dispatch.rs:

• Resolution pipeline: Runs after model_aliases, smart routing, the request-parameter policy snapshot, and the per-key model gate (all of which evaluate the client-supplied model name):
  1. Resolves any model experiment (src/proxy/model_experiments/): assigns the client-facing name to a weighted variant via rendezvous hashing over a SHA-256 sticky key, continues under the variant's upstream model, and narrows the request's backend allow-list if the variant pins a backend.
  2. Applies metadata alias dispatch to the resulting name.
• Return contract (DispatchModel): Carries the client-facing name, the dispatched name, and the experiment assignment so selection errors and fallback attribution keep reporting what the client sent, and a fallback: within_variant experiment can suppress cross-model fallback chains.
• Request-scoped attribution: Uses a tokio::task_local! cell installed by model_experiment_middleware around the API route group (the same mechanism as the upstream TTFT scope). The dispatch seam records the assignment, the response-cache route identity folds it into the exact, prefix, and semantic cache namespaces, and the middleware records model_experiment_* metrics, emits opt-in attribution headers, and applies response_model: client once the handler returns.

Guardrail Layer¶

When guardrails are configured (guardrails.enabled: true), a content-safety layer (GuardrailService in src/services/guardrail/) wraps backend calls across OpenAI chat completions (including the responses_only bridge to upstream /v1/responses), /v1/responses, legacy /v1/completions, and native Anthropic Messages. Stage coverage by route is tabulated in Guardrails: Coverage by route. When guardrails are unconfigured, handlers bypass this seam with zero overhead.

• Unified gating seam and wire views: Driven through src/services/guardrail/gate.rs so every dispatch path shares the same policy and block renderers. Each wire defines its own block shape (Wire::OpenAi, Wire::Responses, Wire::Completions, Wire::Anthropic) and adjacent request/reply views (responses_input_view.rs, responses_view.rs, completions_view.rs, anthropic_input_view.rs).
• Wire-neutral input masking (masking.rs): Each wire enumerates inspectable strings with their JSON locations. Masking verdicts are re-derived per string by the providers that returned them, written back in place (token by token for tool-call arguments in masking_args.rs), and re-checked before dispatch within a deadline, failing closed if masking cannot be applied or a provider fails afterward. Scoped passes run through GuardrailService::run_input_pass (service_pass.rs).
• Terminal block responses: Every block on every wire is rendered by response::block_into_response, which attaches a process-local GuardrailBlocked marker. Both the ingress fallback driver and the /v1/decisions chain (TriggerReason::GuardrailBlocked) treat marked responses as committed so a guardrail block never hops to a fallback model.
• Dynamic service slot (GuardrailServiceHandle): Stored in AppState behind a lock-free, late-settable arc-swap slot (src/services/guardrail/handle.rs) rather than a startup-frozen Option. Request paths load the handle once per request with a single atomic load and reuse it across gates. A router started with guardrails disabled can materialize a service later via guardrail_service_factory (src/server/state.rs, including backend: provider resolution) on a local enabled: true hot-reload or an enforceable Hub guardrail policy. If a Hub policy demands enforcement the router cannot execute, it is rejected whole with guardrails_unavailable on the control-plane policy store while local configuration continues serving.

The layer hooks the request lifecycle at two points:

• Input pre-call: Before backend dispatch, once per request, awaiting all input providers. A Block verdict short-circuits the request without calling the backend; a Transform rewrites the prompt before dispatch. On OpenAI chat completions, it runs once for both streaming and non-streaming arms (src/proxy/chat_input_gate.rs) before response-cache lookup, global prompt injection, and disaggregated dispatch, so a blocked streaming request receives a standard non-streamed block response before SSE starts. (GuardrailService::run_input_during_call remains available for library callers.)
• Output post-call: After the backend returns, over the generated text, before the response is returned or cached. For streaming responses, this runs as a rolling-window check (buffer_full, chunked, or passthrough).

GuardrailService owns all policy (monitor vs. enforce mode, bypass allowlist, per-route provider subsetting, per-category thresholds, allow/deny lists, per-provider timeouts, and fail-open/fail-closed disposition), aggregates verdicts from configured providers (OpenAI Moderation, self-hosted classifier, PII, AWS Bedrock, Azure Content Safety) using a most-severe-wins rule, and emits Prometheus metrics and an audit-log entry for every decision.

sequenceDiagram
    participant Client
    participant Gate as Guardrail Gate
    participant Service as GuardrailService
    participant Backend as LLM Backend

    Client->>Gate: request (prompt / messages)
    Gate->>Service: input pre-call
    alt Block (enforce)
        Service-->>Client: block response (content_filter / refusal / error)
    else Allow or Transform
        Service-->>Gate: verdict
        Gate->>Backend: forward (possibly transformed) request
        Backend-->>Gate: response
        Gate->>Service: output post-call
        alt Block (enforce)
            Service-->>Client: block response
        else Allow or Transform
            Service-->>Client: response (possibly sanitized)
        end
    end

For the operator-facing guide (concepts, provider setup, configuration, threshold tuning, admin controls, and metrics), see Guardrails.

Claude Code Safeguards Layer¶

When safeguards.action_safety.enabled: true is configured, ActionSafetyEvaluator (src/services/action_safety.rs) evaluates tool actions proposed by conversion-backend responses before the client acts on them. When the section is absent, safeguard-bearing requests pass through unchanged on native paths and synthesize nothing on conversion paths.

• Action extraction and evaluation: Driven from the Anthropic Messages handlers and the /v1/responses conversion arm:
  • Extracts proposed tool_use blocks (or /v1/responses tool_calls) in wire order and dispatches once per action to the pinned evaluator backend.
  • Attaches verdicts as safeguard_results via action_safety::wire: a dangerous_tool_use check keyed by tool-use id (not_flagged / flagged) plus an unavailable check when any action resolves indeterminate. On streaming /v1/responses, the results event is emitted before [DONE]; on streaming Anthropic Messages, the terminal message_delta and message_stop are held back within the per-request action budget until pending evaluations complete, and the verdicts ride on that message_delta as delta.safeguard_results (#2682). A reply that proposes no tool use carries an available check with an empty tool_uses map.
  • System One evaluator (safeguards.action_safety.evaluator: systemone): Uses SystemOneEvaluator (src/services/action_safety/systemone.rs) over the shared SystemOneClient (cached per router) to issue one decision call per action over {tool, arguments, user_request, recent_tool_results}. In advisory mode, it runs concurrently with the chat-model evaluator and feeds diagnostics and safeguards_action_safety_advisory_total; in enforce mode, its fail-closed mapping produces the emitted verdict.
  • Fail-closed guarantee: Evaluator unavailability, timeouts, malformed/unknown verdicts, refusals, and budget exhaustion all resolve to non-allow outcomes, never a synthesized allow. Evaluator usage is attributed separately from generation, and diagnostics log only evaluation outcomes (never payloads, tool arguments, or credentials).
• Client-facing POST /v1/decisions endpoint (alias /v1/systemone): Exposes the System One decision protocol outward (src/proxy/decisions/):
  • Validates the request body with shared question rules before dispatch, applies the per-key model gate, and walks fallback.fallback_chains via FallbackService::execute_with_fallback_snapshot.
  • Target passthrough vs. chat emulation: Each hop is either a passthrough to decisions.targets[] via SystemOneClient cached in ClientCache by router and target name (src/services/decisions/targets.rs), or an emulation on a chat model (src/services/decisions/emulation.rs). Emulation converts the question map into a forced emit_decision tool call, applies chat's composed request_params policy and input guardrail pre-call gate (src/proxy/decisions/emulation_gate.rs), dispatches through the chat retry loop (proxy::request::execute_decision_chat_hop), and maps the resulting tool arguments back to one-hot answers marked emulated: true.
  • Routing and access control: Resolves emulated model names via dispatch_model_for_request (metadata aliases and model experiments), rejects inline-endpoint targets for API keys with a non-empty allowed_backends (treated as an unavailable backend on chain hops so the chain skips it without sending), and records usage on the caller's key under the serving model via the non-streaming stats tap.

For the operator-facing guide (native passthrough, adapter setup, client-side workarounds, and the protocol record), see Integrating with Claude Code.

Responses-API Routing for responses_only Models¶

Some OpenAI Pro models (currently gpt-5-pro, gpt-5.2-pro, gpt-5.4-pro, gpt-5.5-pro) are exposed exclusively via /v1/responses and return 404 not_found from /v1/chat/completions. The responses_only: true capability flag in model-metadata.yaml (and the built-in OpenAI registry) marks these models so the router transparently dispatches them to the Responses API surface, regardless of which client surface the request arrived on.

The flag is consulted at the proxy layer right after model lookup, before backend dispatch. The same model metadata pipeline that resolves capability information also resolves responses_only, so the lookup composes with backend-specific overrides, the external metadata file, and built-in defaults in the standard priority order (see Metadata Priority and Alias Resolution).

sequenceDiagram
    participant Client
    participant ChatHandler as POST /v1/chat/completions
    participant AnthropicHandler as POST /anthropic/v1/messages
    participant Lookup as is_responses_only_model()
    participant Converter as Request Converter
    participant Backend as OpenAI / Azure OpenAI

    alt OpenAI surface
        Client->>ChatHandler: chat.completion request
        ChatHandler->>Lookup: (backend, model)
    else Anthropic surface
        Client->>AnthropicHandler: messages request
        AnthropicHandler->>Lookup: (backend, model)
    end

    alt responses_only = true
        alt Backend is OpenAI / Azure OpenAI
            Lookup-->>Converter: Convert to /v1/responses payload
            Converter->>Backend: POST /v1/responses
            Backend-->>Converter: Responses API response (or SSE)
            Converter-->>Client: Translated to original surface shape
        else Backend is not OpenAI / Azure OpenAI
            Lookup-->>Client: 400 invalid_request_error
        end
    else responses_only = false
        Lookup-->>Backend: Standard /v1/chat/completions dispatch
        Backend-->>Client: chat.completion / Anthropic Messages
    end

Key properties:

• The dispatch is transparent on both surfaces: clients keep using /v1/chat/completions or /anthropic/v1/messages and receive responses in the matching shape.
• Only OpenAI and Azure OpenAI backends serve /v1/responses; other backend types are rejected with 400 invalid_request_error before any upstream call.
• The first dispatch per (backend, model) pair logs at info level so operators can confirm Responses-API routing without enabling debug logs.
• Streaming requests are bridged event-by-event: Responses API SSE events are translated into chat.completion.chunk (OpenAI surface) or Anthropic Messages SSE events (Anthropic surface) without buffering the whole response.

For configuration syntax, the current model list, and guidance on marking new models, see Responses-API-only Models in the configuration guide.

Health Check Flow¶

sequenceDiagram
    participant HealthService as Health Service
    participant BackendPool as Backend Pool
    participant Backend as LLM Backend
    participant Cache as Health Cache

    loop Every Interval
        HealthService->>BackendPool: Get All Backends
        BackendPool-->>HealthService: Backend List

        par For Each Backend
            HealthService->>Backend: GET /v1/models (or /health)
            alt Success
                Backend-->>HealthService: 200 OK + Model List
                HealthService->>Cache: Update: consecutive_successes++
                HealthService->>HealthService: Mark Healthy if threshold met
            else Failure
                Backend-->>HealthService: Error/Timeout
                HealthService->>Cache: Update: consecutive_failures++
                HealthService->>HealthService: Mark Unhealthy if threshold met
            end
        end

        HealthService->>BackendPool: Update Backend Health Status
    end

Hot Reload Service¶

Location: src/infrastructure/config/hot_reload.rs, src/infrastructure/config/hot_reload_service.rs

Purpose: Provide runtime configuration updates without server restart

The hot reload system enables zero-downtime configuration changes through automatic file watching and selective component updates.

Key Architecture Components¶

• ConfigManager: File system watching using notify crate, publishes updates via tokio::sync::watch channel
• HotReloadService: Computes configuration differences, classifies changes (immediate/gradual/restart)
• Component Updates: Interior mutability patterns (RwLock) for atomic updates to HealthChecker, CircuitBreaker, RateLimitStore, BackendPool

Backend Pool Hot Reload¶

The BackendPool uses interior mutability (Arc<RwLock<Vec<Arc<Backend>>>>) to support runtime backend additions and removals without service interruption.

Backend States:

• Active: Normal operation, accepts new requests
• Draining: Removed from config, existing requests continue, no new requests
• Removed: Fully cleaned up after all references released

Graceful Draining Process:

1. When a backend is removed from config, it's marked as Draining
2. New requests skip draining backends
3. Existing in-flight requests/streams continue uninterrupted (Arc reference counting)
4. Background cleanup task runs every 10 seconds
5. Backends are removed when: no references remain OR 5-minute timeout exceeded

This ensures zero impact on ongoing connections during configuration changes.

Change Classification¶

• Immediate update: active rate-limit and circuit-breaker policies, global prompts, request parameters, streaming/smart-routing snapshots, selection strategy, prefix routing, the retry policy, per-request timeout budgets, and documented dynamic fields
• Gradual update: backends and health checks
• Requires restart: server/CORS, logging and tracing, the HTTP client's connection and overall timeouts, cache construction, and other startup-built services

Admin API: /admin/config/hot-reload-status for inspecting hot reload capabilities

For detailed hot reload configuration, process flow, and usage examples, see the Hot Reload section in the configuration guide.

Configuration Migration System¶

Location: src/infrastructure/config/{migrator,migration,migrations,fixer,backup}.rs

Purpose: Automatically detect and fix configuration issues, migrate schemas, and ensure configuration validity

The configuration migration system handles configuration evolution and maintenance. It automatically: - Detects and migrates outdated schema versions - Fixes common syntax errors in YAML/TOML files - Validates and corrects configuration values - Creates backups before making changes - Provides dry-run capability for previewing changes

Architecture Components¶

1. Migration Orchestrator (migrator.rs) - Main entry point for migration operations - Coordinates the entire migration workflow - Manages backup creation and restoration - Implements security validations (path traversal, file size limits)

2. Migration Framework (migration.rs) - Defines core types and traits for migrations - Migration trait for implementing version upgrades - ConfigIssue enum for categorizing problems - MigrationResult for tracking changes

3. Schema Migrations (migrations.rs) - Concrete migration implementations (e.g., V1ToV2Migration) - Transforms configuration structure between versions - Example: Converting backend_url to backends array

4. Auto-Correction Engine (fixer.rs) - Detects and fixes common configuration errors - Duration format correction (e.g., "10 seconds" → "10s") - URL validation and protocol addition - Field deprecation handling

5. Backup Manager (backup.rs) - Creates timestamped backups before modifications - Implements resource limits (10MB per file, 100MB total, max 50 backups) - Automatic cleanup of old backups - Preserves file permissions

Migration Workflow¶

graph TD
    A[Read Config File] --> B[Validate Path & Size]
    B --> C[Create Backup]
    C --> D[Parse Configuration]
    D --> E{Parse Success?}
    E -->|No| F[Fix Syntax Errors]
    F --> D
    E -->|Yes| G[Detect Schema Version]
    G --> H{Needs Migration?}
    H -->|Yes| I[Apply Migrations]
    H -->|No| J[Validate Values]
    I --> J
    J --> K{Issues Found?}
    K -->|Yes| L[Apply Auto-Fixes]
    K -->|No| M[Return Config]
    L --> N[Write Updated Config]
    N --> M

Security Features¶

1. Path Traversal Protection: Validates paths to prevent directory traversal attacks
2. File Size Limits: Maximum 10MB configuration files to prevent DoS
3. Format Validation: Only processes .yaml, .yml, and .toml files
4. System Directory Protection: Blocks access to sensitive system paths
5. Test Mode Relaxation: Uses conditional compilation for test-friendly validation

Example Migration: v1.0 to v2.0¶

// V1ToV2Migration implementation
fn migrate(&self, config: &mut Value) -> Result<(), MigrationError> {
    // Convert single backend_url to backends array
    if let Some(backend_url) = config.get("backend_url") {
        let mut backends = Vec::new();
        let mut backend = Map::new();
        backend.insert("url".to_string(), backend_url.clone());

        // Move models to backend
        if let Some(model) = config.get("model") {
            backend.insert("models".to_string(), 
                Value::Sequence(vec![model.clone()]));
        }

        backends.push(Value::Mapping(backend));
        config["backends"] = Value::Sequence(backends);

        // Remove old fields
        config.remove("backend_url");
        config.remove("model");
    }
    Ok(())
}

Configuration Loading Flow¶

graph TD
    A[Application Start] --> B[Config Manager Init]
    B --> C{Config File Specified?}
    C -->|Yes| D[Load Specified File]
    C -->|No| E[Search Standard Locations]

    E --> F{Config File Found?}
    F -->|Yes| G[Load Config File]
    F -->|No| H[Use CLI Args + Env Vars + Defaults]

    D --> I[Parse YAML]
    G --> I
    H --> J[Create Config from Args]

    I --> K[Apply Environment Variable Overrides]
    J --> K

    K --> L[Apply CLI Argument Overrides]
    L --> M[Validate Configuration]
    M --> N{Valid?}
    N -->|Yes| O[Return Config]
    N -->|No| P[Exit with Error]

    O --> Q[Start File Watcher for Hot Reload]
    Q --> R[Application Running]

    Q --> S[Config File Changed]
    S --> T[Reload and Validate]
    T --> U{Valid?}
    U -->|Yes| V[Apply New Config]
    U -->|No| W[Log Error, Keep Old Config]

    V --> R
    W --> R

Dependency Injection¶

Nothing is resolved from a service container at runtime. Services are ordinary constructor arguments: src/server/state.rs builds them during startup, AppState (src/lib.rs) holds them, and each component receives the handles it needs. Where a consumer must not be bound to one concrete implementation, that argument is a trait object rather than a concrete type.

BackendService (src/core/traits.rs) is the seam where this matters. It exposes get_backend, list_backends, get_healthy_backends, and get_stats over the router's backends, and BackendManager (src/infrastructure/backends/pool/backend_manager.rs) is its only production implementation, delegating to a BackendPool and an optional Arc<dyn HealthChecker>. BackendPool is Arc-backed, so BackendManager::from_pool wraps the running pool instead of copying it.

src/core/container.rs still carries a Container from an earlier design that was never wired up: it is constructed only inside its own unit tests and registers no service.

Service Construction¶

Two sites in src/server/state.rs wrap the shared pool as a BackendService and hand it to a consumer.

// src/server/state.rs: the guardrail service factory.
pub(crate) fn guardrail_service_factory(
    client: reqwest::Client,
    backend_pool: BackendPool,
    topology: LiveBackendTopology,
) -> GuardrailServiceFactory {
    Arc::new(move |g_cfg| {
        // BackendManager is the only production BackendService. from_pool
        // shares the router's live pool rather than rebuilding it.
        let backend_resolver: Arc<dyn BackendService> =
            Arc::new(BackendManager::from_pool(backend_pool.clone()));

        // The resolver travels bundled with a live view of the configured
        // backend topology, so a `backend:` guard-model reference is checked
        // against the visibility contract on every guard call.
        let guardrail_service = GuardrailService::from_config(
            g_cfg,
            client.clone(),
            Some(GuardBackendAccess::new(backend_resolver, topology.clone())),
        );
        Arc::new(guardrail_service)
    })
}

// src/server/state.rs: the smart-routing LLM classifier receives the same
// kind of resolver, so classification dispatches through production backends,
// plus a live view of each backend's `request_extensions`, so a classification
// call carries the same extension headers user traffic does.
if let Some(llm_classifier) = LlmClassifier::from_runtime_config(
    config,
    classifier_config,
    Arc::new(BackendManager::from_pool(backend_pool.clone())),
    backend_topology.extensions_source(),
) {
    router = router.with_llm_classifier(llm_classifier, classifier_config.clone());
}

Service Dependencies¶

A consumer keeps the trait object it was handed and calls the trait methods on it. Nothing is looked up.

// src/services/smart_routing/llm_classifier.rs
pub struct LlmClassifier {
    backend_resolver: Arc<dyn BackendService>,
    extensions_source: BackendExtensionsSource,
    backend_name: String,
    // ... prompt, schema, cache, and config fields
}

impl LlmClassifier {
    async fn send_request(&self, request_payload: &Value) -> Result<Value, String> {
        // The backend's current extension headers, then the recursion guard,
        // which replaces any configured spelling of it.
        let headers = self.dispatch_headers();

        let backend = self
            .backend_resolver
            .get_backend(&self.backend_name)
            .await
            .map_err(|e| {
                format!(
                    "classifier backend '{}' is not available in the live backend pool: {e}",
                    self.backend_name
                )
            })?;

        // Dispatching through the resolved backend reuses the same request
        // transformation, authentication, and resilience as user traffic.
        let response = backend
            .execute_chat_completion(request_payload, Some(headers))
            .await;
        // ...
    }
}

Benefits¶

• Testability: a consumer takes Arc<dyn BackendService>, so a test can hand it a BackendManager over a pool it built itself instead of starting the router.
• Shared live state: from_pool shares the running pool's backends, stats, and lifecycle state, so a subsystem observes what production traffic observes rather than a second copy.
• Substitutable resolution: consumers depend on the trait, not on BackendManager, so a different resolver can be swapped in at the construction site without touching them.

Error Handling Strategy¶

The router implements an error handling strategy with typed errors, automatic recovery, and OpenAI-compatible error responses.

Error Type Hierarchy¶

• CoreError: Domain-level errors (validation, service failures, timeouts, configuration)
• RouterError: Application-level errors combining Core, HTTP, Backend, and Model errors
• HttpError: HTTP-specific errors (400 BadRequest, 401 Unauthorized, 404 NotFound, 500 InternalServerError, etc.)

Error Handling Principles¶

1. Fail Fast: Validate inputs early with clear error messages
2. Error Context: Include relevant context (field names, operation details)
3. Retryable Classification: Distinguish between retryable (timeout, 503) and non-retryable (400, 401) errors
4. User-Friendly Responses: Convert internal errors to OpenAI-compatible error format
5. Structured Logging: Log errors with appropriate severity and context

Error Recovery Mechanisms¶

• Retry with Exponential Backoff: Automatically retry transient failures
• Model Fallback: Route to alternative models when primary unavailable (see Model Fallback System)
• Graceful Degradation: Continue with reduced functionality when components fail

For detailed error handling, recovery strategies, monitoring, and troubleshooting, see error-handling.md.

Extension Points¶

Backend Type Architecture¶

The router supports multiple backend types with different API formats. Each backend type handles request/response transformation automatically.

Supported Backend Types¶

Anthropic Backend Architecture¶

The Anthropic backend provides native support for Claude models with automatic format translation:

Key Transformations¶

Request Format Differences:

Extended Thinking Support:

// OpenAI reasoning_effort → Anthropic thinking
{
  "reasoning_effort": "high"  // OpenAI format
}
// Transforms to:
{
  "thinking": {
    "type": "enabled",
    "budget_tokens": 32768   // Mapped from effort level
  }
}

Tool Calling Transformation¶

The router provides automatic transformation of OpenAI-format tool definitions to backend-native formats, enabling cross-provider tool calling support.

Location: src/core/tool_calling/

Module Structure:

src/core/tool_calling/
├── mod.rs           # Module exports
├── definitions.rs   # Type definitions (ToolDefinition, FunctionDefinition, etc.)
├── transform.rs     # Transformation functions for each backend
├── tool_choice.rs   # Tool choice transformation for each backend
├── response.rs      # Tool call response extraction from backends
├── streaming.rs     # Streaming tool call transformation
└── messages.rs      # Multi-turn conversation message transformation

Supported Backends¶

OpenAI Tool Format (Input)¶

{
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": { "type": "string" }
          },
          "required": ["location"]
        }
      }
    }
  ]
}

Anthropic Transformation¶

// transform_tools_to_anthropic()
// Input:  OpenAI format with `function.parameters`
// Output: Anthropic format with `input_schema`
{
  "tools": [
    {
      "name": "get_weather",
      "description": "Get current weather for a location",
      "input_schema": {
        "type": "object",
        "properties": {
          "location": { "type": "string" }
        },
        "required": ["location"]
      }
    }
  ]
}

Gemini Transformation¶

// transform_tools_to_gemini()
// Input:  OpenAI format
// Output: Gemini nested functionDeclarations format
{
  "tools": [
    {
      "functionDeclarations": [
        {
          "name": "get_weather",
          "description": "Get current weather for a location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": { "type": "string" }
            },
            "required": ["location"]
          }
        }
      ]
    }
  ]
}

Validation Rules¶

• Tool Name: Must match pattern ^[a-zA-Z0-9_-]+$ (alphanumeric, underscore, hyphen only)
• Empty Name: Rejected with validation error
• Missing Fields: Logged as warnings, tool skipped (no silent data loss)
• Type Field: Must be "function" for standard tool definitions

Integration with Fallback System¶

The tool calling transformation integrates with the model fallback system via ParameterTranslator:

// In src/core/fallback/translation.rs
impl ParameterTranslator {
    pub fn translate_tools(
        &self,
        tools: &Value,
        from_backend_type: BackendType,
        to_backend_type: BackendType,
    ) -> CoreResult<Value> {
        match (from_backend_type, to_backend_type) {
            (_, BackendType::Anthropic) => transform_tools_to_anthropic(tools),
            (_, BackendType::Gemini) => transform_tools_to_gemini(tools),
            (_, BackendType::LlamaCpp) => transform_tools_for_llamacpp(tools),
            _ => Ok(tools.clone()),
        }
    }
}

This enables fallback from one provider to another while preserving tool definitions.

Tool Choice Transformation¶

In addition to tool definitions, the router also transforms the tool_choice parameter which controls model tool-calling behavior.

Supported Values:

Edge Cases:

• Anthropic "none" workaround: Anthropic API does not support tool_choice=none. When this value is detected, the router removes tools and tool_choice entirely from the request.
• llama.cpp: Preserves parallel_tool_calls parameter for parallel function calling support.

Implementation:

// In src/core/tool_calling/tool_choice.rs
pub enum ToolChoiceValue {
    Auto,
    None,
    Required,
    Function(String),
}

impl ToolChoiceValue {
    pub fn from_openai(value: &Value) -> CoreResult<Self>;
    pub fn to_anthropic(&self) -> Option<Value>;  // Returns None for "none"
    pub fn to_gemini(&self) -> Value;
}

Streaming Tool Call Transformation¶

When streaming is enabled, tool calls are returned incrementally as delta events rather than complete objects. Different providers use distinct streaming formats, requiring real-time transformation to maintain OpenAI API compatibility.

Location: src/core/tool_calling/streaming.rs

Key Components:

Streaming Format Comparison:

State Machine (Anthropic):

IDLE → message_start → READY
READY → content_block_start (tool_use) → TOOL_STARTED
TOOL_STARTED → content_block_delta → ACCUMULATING
ACCUMULATING → content_block_delta → ACCUMULATING (loop)
ACCUMULATING → content_block_stop → TOOL_COMPLETE
TOOL_COMPLETE → message_delta → FINISHED

Safety Limits:

To prevent memory exhaustion from malformed streams:

• MAX_TOOL_CALLS = 64: Maximum parallel tool calls per message
• MAX_ARGUMENTS_SIZE = 1MB: Maximum accumulated arguments per tool call

Example Transformation (Anthropic → OpenAI):

// Anthropic input: content_block_start
{"type": "content_block_start", "index": 0,
 "content_block": {"type": "tool_use", "id": "toolu_123", "name": "get_weather"}}

// OpenAI output: delta chunk
{"id": "chatcmpl-...", "choices": [{"delta": {"tool_calls": [
  {"index": 0, "id": "toolu_123", "type": "function",
   "function": {"name": "get_weather", "arguments": ""}}
]}}]}

Finish Reason Mapping:

Multi-Turn Conversation Message Transformation¶

When using tools in multi-turn conversations, the message history contains tool calls from the assistant and tool results from the client. These messages require format transformation when routing to different backends.

Location: src/core/tool_calling/messages.rs

Key Functions:

Message Format Differences:

OpenAI Format (Input):

{
  "messages": [
    {"role": "user", "content": "What's the weather in NYC?"},
    {
      "role": "assistant",
      "tool_calls": [{
        "id": "call_abc123",
        "type": "function",
        "function": {"name": "get_weather", "arguments": "{\"location\": \"NYC\"}"}
      }]
    },
    {
      "role": "tool",
      "tool_call_id": "call_abc123",
      "content": "{\"temperature\": 72, \"condition\": \"sunny\"}"
    }
  ]
}

Anthropic Transformation:

{
  "messages": [
    {"role": "user", "content": "What's the weather in NYC?"},
    {
      "role": "assistant",
      "content": [{
        "type": "tool_use",
        "id": "call_abc123",
        "name": "get_weather",
        "input": {"location": "NYC"}
      }]
    },
    {
      "role": "user",
      "content": [{
        "type": "tool_result",
        "tool_use_id": "call_abc123",
        "content": "{\"temperature\": 72, \"condition\": \"sunny\"}"
      }]
    }
  ]
}

Gemini Transformation:

{
  "contents": [
    {"role": "user", "parts": [{"text": "What's the weather in NYC?"}]},
    {
      "role": "model",
      "parts": [{
        "functionCall": {"name": "get_weather", "args": {"location": "NYC"}}
      }]
    },
    {
      "role": "function",
      "parts": [{
        "functionResponse": {
          "name": "get_weather",
          "response": {"temperature": 72, "condition": "sunny"}
        }
      }]
    }
  ]
}

Multiple Tool Results:

When multiple tools are called in parallel, consecutive tool result messages are combined for Anthropic into a single user message:

// Multiple OpenAI tool results:
{"role": "tool", "tool_call_id": "call_1", "content": "72F, sunny"}
{"role": "tool", "tool_call_id": "call_2", "content": "3:00 PM EST"}

// Combined Anthropic format:
{
  "role": "user",
  "content": [
    {"type": "tool_result", "tool_use_id": "call_1", "content": "72F, sunny"},
    {"type": "tool_result", "tool_use_id": "call_2", "content": "3:00 PM EST"}
  ]
}

Error Handling:

• Missing tool_call_id: Returns validation error
• Malformed tool calls: Logged as warnings, skipped without failing entire request
• Unknown function name for Gemini: Falls back to "unknown" with warning log
• Malformed JSON arguments: Falls back to empty object {}
• is_error indicator: Preserved when transforming to Anthropic format

Gemini 3 thoughtSignature Support¶

Gemini 3 function calls use a thoughtSignature field that must be preserved and passed back in multi-turn conversations. The router handles this through the transformation pipeline.

What is thoughtSignature?

When Gemini 3+ models return function calls, they include an encrypted thoughtSignature field that encapsulates the model's internal reasoning context. This signature must be included when sending the function result back to continue the conversation correctly.

Response Extraction (Gemini -> Client):

The router extracts thoughtSignature from Gemini responses and includes it in the OpenAI-compatible format:

// Gemini native response (with thoughtSignature)
{
  "candidates": [{
    "content": {
      "parts": [{
        "functionCall": {"name": "get_weather", "args": {"location": "NYC"}},
        "thoughtSignature": "encrypted_signature_abc123"
      }]
    }
  }]
}

// Router's OpenAI-compatible response
{
  "choices": [{
    "message": {
      "tool_calls": [{
        "id": "call_xyz789",
        "type": "function",
        "function": {"name": "get_weather", "arguments": "{\"location\":\"NYC\"}"},
        "extra_content": {
          "google": {
            "thought_signature": "encrypted_signature_abc123"
          }
        }
      }]
    }
  }]
}

Request Injection (Client -> Gemini):

When clients send tool results back, the router extracts thought_signature from extra_content.google and injects it as thoughtSignature in the Gemini request:

// Client's OpenAI-format request
{
  "messages": [{
    "role": "assistant",
    "tool_calls": [{
      "id": "call_xyz789",
      "function": {"name": "get_weather", "arguments": "{}"},
      "extra_content": {
        "google": {"thought_signature": "encrypted_signature_abc123"}
      }
    }]
  }]
}

// Transformed Gemini request
{
  "contents": [{
    "role": "model",
    "parts": [{
      "functionCall": {"name": "get_weather", "args": {}},
      "thoughtSignature": "encrypted_signature_abc123"
    }]
  }]
}

Streaming Support:

The streaming transformer also handles thoughtSignature extraction, including it in tool call delta chunks.

Compatibility Behavior:

• Gemini 2.x models do not return thoughtSignature - the router gracefully handles this
• Clients that don't preserve extra_content.google.thought_signature can still make function calls (but may have degraded conversation continuity on Gemini 3+)
• The extra_content field is ignored by non-Gemini backends

Implementation Locations:

Adding New Backend Types¶

1. Declare the type: add a variant to BackendTypeConfig in src/core/config/models/backend_type.rs. The type: field of each backends[] entry deserializes into this enum. A new variant also requires the configuration-assistant updates listed under "Configuration Assistant Sync" in AGENTS.md.

2. Implement the Backend trait: the trait is defined in src/core/traits.rs and the factory imports it as BackendTrait.

   // In src/infrastructure/backends/custom_backend.rs
   use std::collections::HashMap;
   use std::sync::Arc;
   use std::time::Duration;
   
   use async_trait::async_trait;
   use serde_json::Value;
   
   use crate::core::CoreError;
   use crate::core::traits::{
       Backend, BackendCapabilities, BackendResponse, BackendType, HealthStatus,
   };
   
   pub struct CustomBackend {
       client: Arc<HttpClient>,
       config: CustomBackendConfig,
   }
   
   #[async_trait]
   impl Backend for CustomBackend {
       fn name(&self) -> &str { /* ... */ }
       fn url(&self) -> &str { /* ... */ }
       fn weight(&self) -> u32 { /* ... */ }
       fn supported_models(&self) -> &[String] { /* ... */ }
       fn supports_model(&self, model: &str) -> bool { /* ... */ }
       fn backend_type(&self) -> BackendType { BackendType::Custom("custom".to_string()) }
       fn capabilities(&self) -> BackendCapabilities { /* ... */ }
       fn metadata(&self) -> &HashMap<String, Value> { /* ... */ }
       fn request_timeout(&self) -> Duration { /* ... */ }
   
       async fn health_check(&self) -> Result<HealthStatus, CoreError> { /* ... */ }
       async fn execute_chat_completion(
           &self,
           request: &Value,
           headers: Option<HashMap<String, String>>,
       ) -> Result<BackendResponse, CoreError> { /* ... */ }
   
       // Also required: execute_completion, execute_streaming_request,
       // get_models, get_model_details, get_statistics, get_current_load,
       // validate_config, transform_request, and transform_response.
   }
   

3. Register in the backend factory: add an arm to BackendFactory::create_backend_from_typed_config in src/infrastructure/backends/factory/backend_factory.rs.

   match config.backend_type {
       // ... existing arms ...
       BackendTypeConfig::Custom => {
           let backend = CustomBackend::from_config(config).await?;
           Ok(Arc::new(backend))
       }
   }
   

Adding New Middleware¶

1. Implement Middleware Trait:

   // In src/http/middleware/custom_middleware.rs
   pub struct CustomMiddleware {
       config: CustomConfig,
   }
   
   impl<S> tower::Layer<S> for CustomMiddleware {
       type Service = CustomMiddlewareService<S>;
   
       fn layer(&self, inner: S) -> Self::Service {
           CustomMiddlewareService { inner, config: self.config.clone() }
       }
   }
   

2. Register in HTTP Router:

   // In src/main.rs
   let app = Router::new()
       .route("/v1/models", get(list_models))
       .layer(CustomMiddleware::new(config.custom))
       .layer(LoggingMiddleware::new())
       .with_state(state);
   

Adding New Cache Types¶

1. Implement the CacheStore trait:

   // In src/infrastructure/cache/your_store.rs
   pub struct YourCacheStore { /* ... */ }
   
   #[async_trait]
   impl CacheStore for YourCacheStore {
       async fn get(&self, key: &str) -> CacheStoreResult<Option<Vec<u8>>> { /* ... */ }
       async fn set(&self, key: &str, value: &[u8], ttl: Duration) -> CacheStoreResult<()> { /* ... */ }
       async fn delete(&self, key: &str) -> CacheStoreResult<()> { /* ... */ }
       async fn clear(&self) -> CacheStoreResult<()> { /* ... */ }
       async fn stats(&self) -> CacheStoreStats { /* ... */ }
   }
   

The RedisCacheStore (src/infrastructure/cache/redis_store.rs) is the shipped shared-state implementation, gated behind the redis-cache Cargo feature. That feature is not in default/full, but official release binaries and container images compile it, so a multi-replica deployment can share the cache and rate-limit counters without a custom build. The supported endpoint topology (one managed endpoint; Sentinel and Cluster out of contract) is defined and enforced in src/core/config/redis_endpoint.rs, which both the response cache and the distributed rate limiter validate against at config load.

1. Inject into ResponseCacheStore:

   let store: Arc<dyn CacheStore> = Arc::new(YourCacheStore::new(/* ... */));
   let cache = ResponseCacheStore::with_store(store, capacity, ttl, max_response_size, max_stream_buffer_size);
   

Adding New Load Balancing Strategies¶

Selection strategies are declared in the SelectionStrategy enum (src/core/models/backend.rs), and selection runs through the BackendPool scorer pipeline (BackendScorer in src/core/scoring.rs).

1. Implement BackendScorer:

// In src/core/scoring.rs (or a new scorer module)
pub struct LeastBusyScorer;

#[async_trait]
impl BackendScorer for LeastBusyScorer {
    fn name(&self) -> &'static str { "least_busy" }

    async fn prepare(&self, _context: &ScoringContext) {
        // Cache data that score() reads synchronously (default implementation is a no-op)
    }

    fn score(&self, context: &ScoringContext, backend: &dyn Backend) -> f64 {
        // Higher values indicate a stronger preference.
        // Return f64::NEG_INFINITY to mark the backend as ineligible.
        /* ... */
    }
}

1. Register the scorer with the pool:

pool.register_scorer(Arc::new(LeastBusyScorer)).await;

The base strategies (RoundRobin, WeightedRoundRobin, LeastLatency, Random, ConsistentHash, and others) are configured through SelectionStrategy (src/core/models/backend.rs); see the Load Balancing Guide for configuration.

Design Decisions¶

Why 4-Layer Architecture?¶

Decision: Use a 4-layer architecture (HTTP → Services → Infrastructure → Core)

Rationale¶

• Clear Separation: Each layer has distinct responsibilities
• Testability: Layers can be tested independently
• Maintainability: Changes in one layer don't affect others
• Flexibility: Easy to swap implementations (e.g., different cache backends)

Trade-offs¶

• ✅ Pros: Clean, maintainable, testable, extensible
• ❌ Cons: More complexity, slight performance overhead
• Verdict: Benefits outweigh costs for a production system

Why Constructor Injection Instead of a DI Container?¶

Decision: Pass services as ordinary constructor arguments; use no runtime service container

Rationale¶

• Compile-time wiring: The service graph is visible in src/server/state.rs and checked at build time
• No runtime resolution failure mode: A missing registration is a compile error, not a startup failure
• Testing: Each component is constructed directly with the mocks and test doubles it needs

Alternatives Considered¶

• Runtime DI container (src/core/container.rs): implemented early, never wired up, and carried as dead code

Why Arc<RwLock> for Shared State?¶

Decision: Use Arc<RwLock<T>> for shared mutable state

Rationale¶

• Reader-Writer Semantics: Multiple readers, exclusive writers
• Performance: Better than Arc<Mutex<T>> for read-heavy workloads
• Safety: Prevents data races at compile time

Alternatives Considered¶

• Arc<Mutex<T>>: Simpler but worse performance for reads
• Channels: Too complex for simple shared state
• Atomic types: Not suitable for complex data structures

Why async/await Throughout?¶

Decision: Use async/await for all I/O operations

Rationale¶

• Performance: Non-blocking I/O allows high concurrency
• Resource Efficiency: Lower memory usage than thread-per-request
• Ecosystem: Rust async ecosystem (Tokio, reqwest, axum) is mature

Trade-offs¶

• ✅ Pros: High performance, low resource usage, good ecosystem
• ❌ Cons: Complexity, learning curve, debugging challenges
• Verdict: Essential for high-performance network services

Why Configuration Hot-Reload?¶

Decision: Support configuration hot-reload using file watching

Rationale¶

• Zero Downtime: Update configuration without restarting
• Operations Friendly: Easy to adjust settings in production
• Development: Faster iteration during development

Implementation¶

• File system watcher detects changes
• Validate new configuration before applying
• Atomic updates to avoid inconsistent state
• Fallback to previous config on validation errors

Performance Considerations¶

Memory Management¶

1. Connection Pooling: Reuse HTTP connections to reduce allocation overhead
2. Smart Caching: LRU eviction prevents unbounded memory growth
3. Arc Cloning: Cheap reference counting instead of deep cloning
4. Streaming: Process responses in chunks to avoid loading large responses into memory

Concurrency¶

1. RwLock for Read-Heavy Workloads: Multiple concurrent readers for backend pool and model cache
2. Lock-Free Where Possible: Use atomics for counters and simple state
3. Async Task Spawning: Background tasks for health checks and cache updates
4. Bounded Channels: Prevent unbounded queuing of tasks
5. Streaming Relay Channels: Client-facing SSE relays are the exception. Every relay sends through the one shared relay channel (services::streaming::sse_relay_channel), which never waits on the client, so the relay task reads the upstream at the upstream's pace and observes the terminal usage however slowly the client reads; a disconnect fails the next send, which stops the relay and drops the upstream. Per-stream buffering is bounded by the upstream reply (its output-token cap and timeouts.streaming.total). tests/streaming_relay_pacing_audit_test.rs rejects a bounded channel of SSE events and an Sse body that does not serve this channel.

I/O Optimization¶

1. Connection Keep-Alive: TCP connections stay open for reuse
2. Streaming Responses: Forward SSE chunks without buffering
3. Timeouts: Prevent hanging on slow backends
4. Retry with Backoff: Avoid overwhelming failing backends

Memory Layout¶

// Optimized data structures for cache efficiency
pub struct Backend {
    pub name: String,          // Inline string for small names
    pub url: Arc<str>,         // Shared string for URL
    pub weight: u32,           // Compact integer
    pub is_healthy: AtomicBool, // Lock-free health status
}

// Cache-friendly model storage
pub struct ModelCache {
    models: HashMap<String, Arc<ModelInfo>>, // Shared model info
    last_updated: AtomicU64,                 // Lock-free timestamp
    ttl: Duration,
}

Benchmarking Results¶

Based on our benchmarks (see benches/performance_benchmarks.rs):

• Request Latency: < 5ms overhead for routing decisions
• Memory Usage: ~50MB base memory, scales linearly with backends
• Throughput: 1000+ requests/second on modest hardware
• Connection Efficiency: 100+ concurrent connections per backend with minimal memory overhead

Rate Limiting¶

The router implements rate limiting to protect against abuse and ensure fair resource allocation across clients.

Key Features: - Dual-window approach: sustained limit (100 req/min) + burst protection (20 req/5s) - Client identification by API key (preferred) or IP address (fallback) - Per-client isolation with automatic cache cleanup - DoS prevention with short TTL for empty responses

Rate Limit V2 Architecture¶

The enhanced rate limiting system (rate_limit_v2/) provides a modular, high-performance implementation:

Module Structure¶

src/http/middleware/rate_limit_v2/
├── mod.rs              # Public API and module exports
├── middleware.rs       # Axum middleware integration
├── store.rs            # Rate limit storage and client tracking
└── token_bucket.rs     # Token bucket algorithm implementation

Components¶

1. Token Bucket Algorithm (token_bucket.rs)
2. Configurable bucket capacity and refill rate
3. Atomic operations for lock-free token consumption
4. Automatic token replenishment based on elapsed time

5. Separate buckets for sustained and burst limits

6. Rate Limit Store (store.rs)

7. Per-client state tracking with DashMap for concurrent access
8. Automatic cleanup of expired client entries
9. Configurable TTL for inactive clients (default: 1 hour)

10. Memory-efficient with bounded storage

11. Middleware Integration (middleware.rs)

12. Extracts client identifier (API key → IP address fallback)
13. Checks both sustained and burst limits before processing
14. Returns HTTP 429 (Too Many Requests) with Retry-After header
15. Prometheus metrics for monitoring rate limit hits

Configuration Example¶

rate_limiting:
  enabled: true
  sustained:
    max_requests: 100
    window_seconds: 60
  burst:
    max_requests: 20
    window_seconds: 5
  cleanup_interval_seconds: 300

Decision Flow¶

Request arrives
    ↓
Extract client ID (API key or IP)
    ↓
Check sustained limit (100 req/min)
    ↓ OK
Check burst limit (20 req/5s)
    ↓ OK
Process request

For detailed configuration information, see the Rate Limiting section in the configuration guide.

Model Fallback System¶

The router implements a configurable model fallback system that automatically routes requests to alternative models when the primary model is unavailable.

Key Features: - Automatic fallback chain execution (e.g., gpt-5.6-sol → gpt-5.6-terra → gpt-5.6-luna) - Cross-provider fallback support with a canonical request payload on every hop - Triggering from configured HTTP errors, timeouts, connection failures, model lookup failures, and backend health - Prometheus metrics for monitoring fallback usage

For detailed configuration and implementation, see error-handling.md section on model fallback.

Circuit Breaker¶

The router contains a per-backend circuit-breaker state machine with Admin inspection and force controls. The ordinary LLM proxy path records request outcomes in it and consults it during selection: an open backend circuit is excluded so traffic routes around it, and when every compatible circuit is open the router returns the standard service-unavailable error and participates in configured fallback. It complements active health checks, retry, and model fallback on that path.

Three-State Machine:

Key Features: - Per-backend isolation with independent state - Lock-free atomic operations for minimal hot-path overhead - Admin endpoints for manual control (/admin/circuit/*) - A metrics collector registered in the main metrics registry and populated by the proxy path

For detailed configuration and implementation, see error-handling.md section on circuit breaker.

File Storage¶

The router provides OpenAI Files API compatible file storage with persistent metadata.

Key Features: - Persistent metadata storage with sidecar JSON files - Automatic recovery on server restart - Orphan file detection and cleanup - Pluggable backends (memory/persistent)

For detailed architecture and implementation, see File Storage Guide.

Image Generation Architecture¶

The router provides a unified interface for image generation across multiple backends (OpenAI GPT Image, DALL-E, and Google Gemini/Nano Banana) with automatic parameter translation.

Multi-Backend Image Generation¶

OpenAI → Gemini Parameter Conversion¶

When using Nano Banana (Gemini) models, OpenAI-style parameters are automatically converted to Gemini's native format:

Size to Aspect Ratio Mapping¶

Request Transformation¶

OpenAI Format (Input):

{
  "model": "nano-banana",
  "prompt": "A serene Japanese garden",
  "size": "1536x1024",
  "n": 1
}

Gemini Format (Converted):

{
  "contents": [
    {
      "parts": [{"text": "A serene Japanese garden"}]
    }
  ],
  "generationConfig": {
    "imageConfig": {
      "aspectRatio": "3:2",
      "imageSize": "1K"
    }
  }
}

Conversion Implementation¶

The conversion is handled by src/infrastructure/backends/gemini/image_generation.rs:

pub fn convert_openai_to_gemini(request: &OpenAIImageRequest)
    -> CoreResult<(String, GeminiImageRequest)>
{
    // 1. Map model name
    let gemini_model = map_model_to_gemini(&request.model);

    // 2. Parse size to aspect ratio and size category
    let parsed_size = parse_openai_size(&request.size, &request.model)?;

    // 3. Build Gemini request with imageConfig
    let gemini_request = GeminiImageRequest {
        contents: vec![GeminiContent { parts: vec![...] }],
        generation_config: Some(GeminiGenerationConfig {
            image_config: Some(GeminiImageConfig {
                aspect_ratio: Some(parsed_size.aspect_ratio.to_gemini_string()),
                image_size: Some(parsed_size.size_category.to_gemini_image_size()),
            }),
        }),
    };

    Ok((gemini_model, gemini_request))
}

Streaming Image Generation (SSE)¶

For GPT Image models, the router supports true SSE passthrough for streaming image generation:

┌─────────────┐                ┌─────────────┐                ┌─────────────┐
│   Client    │────stream:true─▶│   Router    │────stream:true─▶│   OpenAI    │
│             │                │             │                │             │
│             │◀───SSE events──│  Passthrough│◀───SSE events──│             │
└─────────────┘                └─────────────┘                └─────────────┘

SSE Event Types:

Implementation (src/proxy/image_gen.rs):

async fn handle_streaming_image_generation(...) -> Result<Response, StatusCode> {
    // 1. Keep stream: true in backend request
    // 2. Make streaming request via bytes_stream()
    // 3. Forward SSE events through tokio channel

    let (tx, rx) = tokio::sync::mpsc::unbounded_channel();

    tokio::spawn(async move {
        let mut stream = backend_response.bytes_stream();
        while let Some(chunk) = stream.next().await {
            // Parse SSE format (event:/data: lines)
            // Forward events to client
            for line in chunk_str.lines() {
                if let Some(event_type) = line.strip_prefix("event:") { ... }
                if let Some(data) = line.strip_prefix("data:") {
                    let event = Event::default().event(event_type).data(data);
                    tx.send(Ok(event));
                }
            }
        }
    });

    Ok(Sse::new(UnboundedReceiverStream::new(rx)).into_response())
}