Architecture Guide¶

This document covers Continuum Router's architecture, design decisions, and extension points.

Table of Contents¶

• Overview
• 4-Layer Architecture
• Library Entry Point (src/server/)
• Core Components
• Data Flow
• Dependency Injection
• Error Handling Strategy
• Extension Points
• Design Decisions
• Performance Considerations
• Rate Limiting → Configuration
• Model Fallback System → Error Handling
• Circuit Breaker → Error Handling
• File Storage → Architecture Details
• Agent Communication Protocol (ACP)
• Backend Passthrough Contract
• Model ID Suffix Normalization → Configuration

Overview¶

Continuum Router is a high-performance, production-ready LLM API router using a 4-layer architecture that provides separation of concerns, testability, and maintainability. The architecture follows Domain-Driven Design principles and dependency inversion to create an extensible system.

Architecture Goals¶

1. Separation of Concerns: Each layer has a single, well-defined responsibility
2. Dependency Inversion: Higher layers depend on abstractions, not concrete implementations
3. Testability: Each component can be unit tested in isolation
4. Extensibility: New features can be added without modifying existing code
5. Performance: Minimal overhead while maintaining clean architecture
6. Reliability: Fail-fast design with full error handling

4-Layer Architecture¶

Layer Descriptions¶

1. HTTP Layer (src/http/)¶

Responsibility: Handle HTTP requests, responses, and web-specific concerns

Components¶

• Routes (routes.rs): Define HTTP endpoints and route handling
• Middleware (middleware/): Cross-cutting concerns (auth, logging, metrics, rate limiting)
• DTOs (dto/): Data Transfer Objects for HTTP serialization/deserialization
• Streaming (streaming/): Server-Sent Events (SSE) handling

Key Files¶

src/http/
├── mod.rs              # HTTP layer exports
├── routes.rs           # Route definitions and handlers
├── dto.rs              # Request/Response DTOs
├── handlers/           # Request handlers
│   ├── mod.rs
│   └── responses.rs    # Responses API handlers
├── middleware/         # HTTP middleware components
│   ├── mod.rs
│   ├── auth.rs         # API key authentication middleware
│   ├── admin_auth.rs   # Admin API authentication middleware
│   ├── files_auth.rs   # Files API authentication middleware
│   ├── admin_audit.rs  # Admin operations audit logging
│   ├── cors.rs         # CORS (Cross-Origin Resource Sharing) middleware
│   ├── logging.rs      # Request/response logging
│   ├── metrics.rs      # Metrics collection
│   ├── metrics_auth.rs # Metrics endpoint authentication
│   ├── model_extractor.rs # Model extraction from requests
│   ├── prometheus.rs   # Prometheus metrics integration
│   ├── rate_limit.rs   # Rate limiting middleware (legacy)
│   └── rate_limit_v2/  # Enhanced rate limiting (modular)
│       ├── mod.rs      # Module exports
│       ├── middleware.rs # Rate limiting middleware
│       ├── store.rs    # Rate limit storage and tracking
│       └── token_bucket.rs # Token bucket algorithm
└── streaming/          # SSE streaming handlers
    ├── mod.rs
    └── handler.rs      # Streaming response handling

Middleware Components¶

The HTTP layer includes several middleware components that provide cross-cutting concerns:

• auth.rs: API key authentication for main endpoints (/v1/chat/completions, /v1/models, /anthropic/*, etc.)

  • Validates API keys presented as Authorization: Bearer <key> or the Anthropic-native x-api-key: <key> header; Authorization: Bearer takes precedence when both are present
  • Supports multiple API keys configured in config.yaml
  • Returns 401 Unauthorized for invalid/missing keys

• admin_auth.rs: Separate authentication for admin endpoints (/admin/*)

  • Supports none, bearer token, HTTP Basic, and IP-whitelist modes
  • Protects sensitive operations (config reload, circuit-breaker control, health management)
  • Configurable through admin.auth; the accepted api_key method is not operational in the standard server because the middleware is constructed without the API-key store

• files_auth.rs: Authentication middleware for Files API (/v1/files/*)

  • Validates API keys specifically for file upload/download/deletion operations
  • Prevents unauthorized file access and manipulation
  • Integrates with file storage service for permission checks

• admin_audit.rs: Audit logging middleware for admin operations

  • Records all admin API calls with timestamps and caller identification
  • Logs parameters and outcomes of sensitive operations
  • Provides audit trail for compliance and security monitoring
  • Configurable log levels and retention policies

• cors.rs: CORS (Cross-Origin Resource Sharing) middleware

  • Enables embedding the router in web applications, Tauri apps, and Electron apps
  • Supports wildcard origins (*), exact origins, and port wildcards (http://localhost:*)
  • Custom scheme support for desktop apps (e.g., tauri://localhost)
  • Configurable methods, headers, credentials, and preflight cache duration
  • Applied early in the middleware stack for proper preflight handling

• rate_limit_v2/: Enhanced rate limiting system (see Rate Limiting section)

  • Token bucket algorithm with per-client tracking
  • Separate limits for sustained rate and burst protection
  • Automatic cleanup of expired client entries
  • Detailed metrics for monitoring

2. Services Layer (src/services/)¶

Responsibility: Orchestrate business logic and coordinate between infrastructure components

Components¶

• Backend Service (backend_service.rs): Manage backend pool, load balancing, health checks
• Model Service (model_service.rs): Aggregate models from backends, handle caching, enrich with metadata
• Proxy Service (proxy_service.rs): Route requests, handle retries, manage streaming
• Health Service (health_service.rs): Monitor service health, track status
• Service Registry (mod.rs): Manage service lifecycle and dependencies
• Web Search Service (search/): Router-managed web_search tool for self-hosted backends. Pluggable provider trait, Serper implementation, bounded tool-execution loop with iteration cap / wall-clock / byte-budget guards (see Web Search)

Key Files¶

src/services/
├── mod.rs              # Service registry and management
├── backend_service.rs  # Backend management service
├── model_service.rs    # Model aggregation service
├── proxy_service.rs    # Request proxying and routing
├── health_service.rs   # Health monitoring service
├── deduplication.rs    # Request deduplication service
├── responses/          # Responses API support
│   ├── mod.rs
│   ├── converter.rs    # Response format conversion
│   ├── router.rs       # Routing strategy determination (pass-through vs conversion)
│   ├── passthrough.rs  # Direct pass-through for native OpenAI/Azure backends
│   ├── session.rs      # Session management
│   ├── stream_service.rs # Streaming service orchestration
│   └── streaming.rs    # Streaming response handling
├── search/             # Router-managed web search service
│   ├── mod.rs          # SearchProvider trait, SearchError, SearchResult, build_provider
│   ├── executor.rs     # WebSearchExecutor: tool injection, tool-execution loop
│   └── providers.rs    # Concrete provider implementations (Serper, Exa stub, Brave stub)
└── streaming/          # Streaming utilities
    ├── mod.rs
    ├── parser.rs       # Stream parsing logic
    └── transformer.rs  # Stream transformation (OpenAI/Anthropic)

3. Infrastructure Layer (src/infrastructure/)¶

Responsibility: Provide concrete implementations of external systems and technical capabilities

Components¶

• Backends (backends/): Specific backend implementations (OpenAI, Anthropic, Gemini, vLLM, Ollama, llama.cpp, MLxcel, LM Studio, Continuum Router)
• Auth (auth/): OAuth 2.0 Device Authorization Grant (RFC 8628) subsystem. The AuthStrategyRegistry is populated at startup and updated during hot-reload; every proxy hot-path handler (chat completions, responses, image generation) consults it to sign outbound requests and force-refresh on 401. See src/proxy/oauth_helper.rs for the shared proxy auth-helper.
• Cache (cache/): Caching implementations (LRU/TTL-based TtlLruCache, CacheStore trait for pluggable backends, InMemoryCacheStore, SHA256-keyed ResponseCacheStore, retry-aware cache)
• KV Cache Index (kv_index/): Shared index that maps token prefix hashes to backends for KV cache-aware routing. Provides InMemoryKvIndex (DashMap + LRU eviction, single-router) and RedisKvIndex (Redis sorted sets, multi-router). Includes KvOverlapScorer which implements BackendScorer to score backends by cache overlap, load, and health. The event consumer ingests real-time cache events from vLLM SSE streams to keep the index current.
• Configuration (config/): Configuration loading, watching, validation
• HTTP Client (http_client.rs): HTTP client management and optimization

Key Files¶

src/infrastructure/
├── mod.rs              # Infrastructure exports and utilities
├── auth/               # OAuth 2.0 Device Authorization Grant (RFC 8628)
│   ├── mod.rs          # Public re-exports
│   ├── device_flow.rs  # RFC 8628 state machine + ReqwestDeviceFlowClient
│   ├── token_store.rs  # Atomic write (tempfile+rename), 0600 perms, ~ expansion
│   ├── strategy.rs     # AsyncAuthStrategy trait + OAuthAuthStrategy (proactive refresh)
│   ├── registry.rs     # AuthStrategyRegistry: hot-reloadable map of backend → strategy
│   └── error.rs        # AuthError type
├── backends/           # Backend implementations
│   ├── mod.rs
│   ├── anthropic/      # Native Anthropic Claude backend
│   │   ├── mod.rs      # Backend implementation & request transformation
│   │   └── stream.rs   # SSE stream transformer (Anthropic → OpenAI)
│   ├── gemini/         # Native Google Gemini backend
│   │   ├── mod.rs      # Backend implementation with TTFB optimization
│   │   └── stream.rs   # SSE stream transformer (Gemini → OpenAI)
│   ├── openai/         # OpenAI-compatible backend
│   │   ├── mod.rs
│   │   ├── backend.rs  # OpenAI backend implementation
│   │   └── models/     # OpenAI-specific model definitions
│   ├── factory/        # Backend factory pattern
│   │   ├── mod.rs
│   │   └── backend_factory.rs  # Creates backends from config
│   ├── pool/           # Backend pooling and management
│   │   ├── mod.rs
│   │   ├── backend_pool.rs     # Connection pool management
│   │   └── backend_manager.rs  # Backend lifecycle management
│   ├── generic/        # Generic backend implementations
│   │   └── mod.rs
│   ├── llamacpp/       # llama.cpp / llama-server backend
│   │   ├── mod.rs
│   │   ├── backend.rs     # LlamaCppBackend implementation
│   │   ├── model_parser.rs # Hybrid response format parsing
│   │   └── props.rs       # /props endpoint for tool calling detection
│   ├── continuum_router/  # Continuum Router / Backend.AI GO backend
│   │   ├── mod.rs
│   │   └── backend.rs     # ContinuumRouterBackend implementation
│   └── vllm.rs         # vLLM/Ollama/LM Studio backend implementation
├── common/             # Shared infrastructure utilities
│   ├── mod.rs
│   ├── executor.rs     # Request execution with retry/metrics
│   ├── headers.rs      # HTTP header utilities
│   ├── http_client.rs  # HTTP client factory with pooling
│   ├── statistics.rs   # Backend statistics collection
│   └── url_validator.rs # URL validation and security
├── transport/          # Transport abstraction layer
│   ├── mod.rs               # Transport enum (HTTP, UnixSocket)
│   ├── unix_socket.rs       # Unix Domain Socket client (Linux/macOS)
│   └── unix_socket_windows.rs  # Windows AF_UNIX client via socket2
├── cache/              # Caching implementations
│   ├── mod.rs
│   ├── store.rs          # CacheStore trait (async, byte-oriented cache backend abstraction)
│   ├── memory_store.rs   # InMemoryCacheStore: TtlLruCache-backed CacheStore implementation
│   ├── lru_cache.rs      # LRU cache with TTL and eviction
│   ├── response_cache.rs # SHA256-keyed LLM response cache (uses Arc<dyn CacheStore>)
│   └── retry_cache.rs    # Retry-aware cache
├── kv_index/           # KV cache prefix-hash index and overlap scoring
│   ├── mod.rs            # Re-exports KvCacheIndex, InMemoryKvIndex, RedisKvIndex, KvOverlapScorer
│   ├── index.rs          # KvCacheIndex trait + InMemoryKvIndex (DashMap/LRU) + RedisKvIndex (sorted sets)
│   ├── event_consumer.rs # KvEventConsumerManager: subscribes to vLLM SSE KV cache event streams
│   ├── scorer.rs         # KvOverlapScorer: BackendScorer impl using KV cache affinity + load + health
│   └── types.rs          # KvCacheEvent, KvCacheEventType, ConsumerStatus, KvEventConsumerConfig
├── config/             # Configuration management
│   ├── mod.rs
│   ├── loader.rs       # Configuration loading
│   ├── validator.rs    # Configuration validation
│   ├── timeout_validator.rs # Timeout configuration validation
│   ├── watcher.rs      # File watching for hot-reload
│   ├── migrator.rs     # Configuration migration orchestrator
│   ├── migration.rs    # Migration types and traits
│   ├── migrations.rs   # Specific migration implementations
│   ├── fixer.rs        # Auto-correction logic
│   ├── backup.rs       # Backup management
│   └── secrets.rs      # Secret/API key management
└── lock_optimization.rs # Lock and concurrency optimization

4. Core Layer (src/core/)¶

Responsibility: Define domain models, business rules, and fundamental abstractions

Components¶

• Models (models/): Core domain entities (Backend, Model, Request, Response)
• Traits (traits.rs): Core interfaces and contracts
• Errors (errors.rs): Domain-specific error types and handling
• Retry (retry/): Retry policies and strategies
• Container (container.rs): Dependency injection container
• Scoring (scoring.rs): BackendScorer trait for composable backend scoring pipeline; each scorer assigns a signal-specific score and BackendPool sums them to select the best backend

Key Files¶

src/core/
├── mod.rs              # Core exports and utilities
├── models/             # Domain models
│   ├── mod.rs
│   ├── backend.rs      # Backend domain model
│   ├── model.rs        # LLM model representation
│   ├── request.rs      # Request models
│   └── responses.rs    # Response models (Responses API)
├── traits.rs           # Core traits and interfaces
├── errors.rs           # Error types and handling
├── container.rs        # Dependency injection container
├── scoring.rs          # BackendScorer trait: composable async scoring pipeline for backend selection
├── hashing.rs          # SHA256 prefix key computation (compute_hash)
├── prefix_key.rs       # Prefix key extraction from request bodies
├── async_utils.rs      # Async utility functions
├── duration_utils.rs   # Duration parsing utilities
├── streaming/          # Streaming models
│   ├── mod.rs
│   └── models.rs       # Streaming-specific models
├── retry/              # Retry mechanisms
│   ├── mod.rs
│   ├── policy.rs       # Retry policies
│   └── strategy.rs     # Retry strategies
├── circuit_breaker/    # Circuit breaker pattern
│   ├── mod.rs          # Module exports
│   ├── config.rs       # Configuration models
│   ├── state.rs        # State machine and breaker logic
│   ├── error.rs        # Circuit breaker errors
│   ├── metrics.rs      # Prometheus metrics
│   └── tests.rs        # Unit tests
├── files/              # File processing utilities
│   ├── mod.rs          # Module exports
│   ├── resolver.rs     # File reference resolution in chat requests
│   ├── transformer.rs  # Message transformation with file content
│   └── transformer_utils.rs # Transformation utility functions
├── tool_calling/       # Tool calling transformation
│   ├── mod.rs          # Module exports
│   ├── definitions.rs  # Tool/function type definitions
│   └── transform.rs    # Backend-specific transformations
└── config/             # Configuration models
    ├── mod.rs
    ├── models/            # Configuration data models (modular structure)
    │   ├── mod.rs         # Public module exports
    │   ├── config.rs      # Main Config struct, ServerConfig, BackendConfig
    │   ├── backend_type.rs # BackendType enum definitions
    │   ├── model_metadata.rs # ModelMetadata, PricingInfo, CapabilityInfo
    │   ├── global_prompts.rs # GlobalPrompts configuration
    │   ├── samples.rs     # Sample generation configurations
    │   ├── validation.rs  # Configuration validation logic
    │   └── error.rs       # Configuration-specific errors
    ├── timeout_models.rs # Timeout configuration models
    ├── cached_timeout.rs # Cached timeout resolution
    ├── optimized_retry.rs # Optimized retry configuration
    ├── metrics.rs      # Metrics configuration
    └── rate_limit.rs   # Rate limit configuration

Library Entry Point (src/server/)¶

Responsibility: Expose Continuum Router as an embeddable Rust library crate via a ContinuumRouter builder API

The src/server/ module decouples all initialization logic from the binary entry point (main.rs), enabling downstream Rust projects to embed the router directly without spawning a subprocess. The binary main.rs becomes a thin CLI wrapper (~220 lines) that delegates to this module.

Key Files:

src/server/
├── mod.rs      # ContinuumRouter struct, ContinuumRouterBuilder with fluent API
├── init.rs     # Individual service initialization functions (HTTP client, health checker,
│               # circuit breaker, file service, API key store, rate limiting)
├── state.rs    # initialize_services() — assembles all services into AppState
├── routes.rs   # Route builders for API, admin, files, WebUI, metrics, and CORS
└── serve.rs    # Server lifecycle: bind listeners, hot-reload, graceful shutdown

Builder API:

use continuum_router::{ContinuumRouter, Config};

// From a config file (enables hot-reload when called)
let router = ContinuumRouter::from_config_file("config.yaml")
    .await?
    .enable_hot_reload(true)
    .build()
    .await?;
router.serve("0.0.0.0:8080").await?;

// From a programmatic Config struct (no file dependency)
let router = ContinuumRouter::from_config(Config::default())
    .enable_health_checks(false)
    .with_config_dir("/etc/continuum")
    .build()
    .await?;

// Embed into an existing Axum application
let app = axum::Router::new()
    .nest("/llm", router.into_router());

Builder Override Semantics:

Each enable_* method registers an explicit override. When None (not called), the builder respects whatever the config file or Config::default() says. When Some(value), the override wins regardless of the config source.

ContinuumRouter struct (returned by build()):

Design Decisions:

• initialize_prompt_file_cache accepts Option<&Path> instead of &Args, removing the CLI dependency from library code.
• perform_container_health_check (Docker HEALTHCHECK) remains in main.rs because it is binary-only and creates no TCP listening sockets.
• load_configuration (CLI fallback logic) also remains in main.rs to keep the library free of CLI concerns.
• All init::* and state::* functions are pub(crate) or pub(super); only ContinuumRouter, ContinuumRouterBuilder, and the init / routes sub-modules are re-exported from lib.rs.

Core Components¶

Backend Pool¶

Location: src/infrastructure/backends/pool/ (backend_pool.rs + lifecycle.rs)

Purpose: The single source of truth for backend membership, selection, and lifecycle. It holds Arc<dyn Backend> execution objects, runs the health-aware selection (round-robin / weighted / least-latency / consistent-hash / prefix-aware, plus the composable scorer pipeline with in-flight counting and per-backend stats), and owns the graceful-drain and hot-reload lifecycle.

Every live HTTP request path (chat completions, streaming chat, Responses, Anthropic Messages and count_tokens, image generation) dispatches its final backend pick through this pool. Each path resolves and pre-filters its own candidate set (model lookup, internal-backend filter, per-key allow-list, retry-state exclusion), then hands the names to the shared selection seam (proxy::selection::select_pooled_backend), which health-filters them and calls BackendPool::select_from_candidates so the configured selection_strategy (and any registered scorers, e.g. the KvOverlapScorer registered when prefix_routing.enabled and a KV index are configured) governs traffic distribution instead of candidate-list order. The consistent-hash ring cache is a small bounded LRU keyed by the participating backend-name vector (including order), so alternating per-model candidate subsets reuse their own rings without unbounded growth. Each selection call clones an Arc ring snapshot before walking it, which prevents a ring built for one candidate set from being interpreted against another and keeps the cache lock out of the ring walk.

pub struct BackendPool {
    backends: Arc<RwLock<Vec<Arc<dyn Backend>>>>,
    counter: Arc<AtomicUsize>,
    selection_strategy: SelectionStrategy,
    stats: Arc<RwLock<HashMap<String, BackendStats>>>,
    scorers: Arc<RwLock<Vec<Arc<dyn BackendScorer>>>>,
    // Lifecycle side tables (keyed by backend name), see lifecycle.rs:
    backend_states: Arc<RwLock<HashMap<String, BackendState>>>, // Active/Draining/Removed
    draining: Arc<RwLock<Vec<DrainingEntry>>>,                  // + drain timestamps
    // ... hash-ring cache, epsilon, scorer threshold ...
}

impl BackendPool {
    // Health/scorer-aware selection over the live backends.
    pub async fn select_backend(&self, model: Option<&str>) -> CoreResult<Arc<dyn Backend>> { /* ... */ }

    // Candidate-restricted selection used by every live HTTP path:
    // intersects the caller's pre-filtered candidate names with pool membership
    // (by name only, preserving order), then runs the scorer pipeline and the
    // configured strategy over exactly that subset.
    pub async fn select_from_candidates(
        &self,
        candidates: &[String],
        model: Option<&str>,
        context: &ScoringContext,
    ) -> CoreResult<Arc<dyn Backend>> { /* ... */ }

    // Graceful drain (out of rotation, in-flight requests finish) + cleanup.
    pub async fn drain_backend(&self, name: &str) -> Option<Arc<dyn Backend>> { /* ... */ }
    pub async fn cleanup_draining(&self) -> usize { /* ... */ }

    // In-place hot reload: add/drain/recreate backends from new config.
    pub async fn update_from_config(&self, configs: &[BackendConfig], factory: &BackendFactory) { /* ... */ }
}

The mutable lifecycle state lives in side tables keyed by backend name (backend_states, draining), mirroring the pool's existing name-keyed stats map, so the hot selection path keeps its plain Vec<Arc<dyn Backend>> layout. AppState holds exactly one backend_pool; the proxy/passthrough layers project the chosen Arc<dyn Backend> to a lightweight PooledBackend { name, url } handle at the selection boundary.

Health Checker¶

Location: src/health.rs → src/services/health_service.rs

Purpose: Monitor backend health with configurable thresholds, automatic recovery, and accelerated warmup detection

pub struct HealthChecker {
    backends: Arc<RwLock<Vec<Backend>>>,
    config: HealthConfig,
    status_map: Arc<RwLock<HashMap<String, HealthStatus>>>,
}

pub struct HealthConfig {
    pub interval: Duration,
    pub timeout: Duration,
    pub unhealthy_threshold: u32,     // Failures before marking unhealthy
    pub healthy_threshold: u32,       // Successes before marking healthy
    pub warmup_check_interval: Duration, // Accelerated interval during warmup (default: 1s)
    pub max_warmup_duration: Duration,   // Max time in warmup mode (default: 300s)
}

pub enum HealthStatus {
    Healthy,      // Backend responding with HTTP 200
    Unhealthy,    // Connection failure or error
    WarmingUp,    // HTTP 503 - backend loading (accelerated checks)
    Unknown,      // Initial state
}

Accelerated Warmup Health Checks¶

When a backend returns HTTP 503 (Service Unavailable), it enters the WarmingUp state. During this state:

• Health checks run at warmup_check_interval (default: 1 second) instead of the normal interval
• This reduces model availability detection from ~30 seconds to ~1 second
• After max_warmup_duration, the backend is marked as Unhealthy
• Particularly useful for llama.cpp backends that return HTTP 503 during model loading

Transport Layer¶

Location: src/infrastructure/transport/

Purpose: Provide a unified transport abstraction for backend communication over HTTP/HTTPS or Unix Domain Sockets

The transport layer enables secure local LLM communication via Unix sockets, eliminating the need for TCP port exposure.

URL Schemes¶

Transport Enum¶

pub enum Transport {
    Http { url: String },
    UnixSocket { socket_path: PathBuf },
}

impl Transport {
    pub fn from_url(url: &str) -> Result<Self, TransportError>;
    pub fn is_unix_socket(&self) -> bool;
    pub fn is_http(&self) -> bool;
}

Unix Socket Client¶

The UnixSocketClient provides HTTP-over-Unix-socket communication:

pub struct UnixSocketClient {
    socket_path: PathBuf,
    config: UnixSocketClientConfig,
}

impl UnixSocketClient {
    pub async fn get(&self, endpoint: &str, headers: Option<Vec<(String, String)>>)
        -> Result<UnixSocketResponse, UnixSocketError>;
    pub async fn post(&self, endpoint: &str, headers: Option<Vec<(String, String)>>, body: Bytes)
        -> Result<UnixSocketResponse, UnixSocketError>;
    pub async fn health_check(&self) -> Result<bool, UnixSocketError>;
}

Security Features¶

• Path Traversal Protection: Validates socket paths to prevent directory traversal attacks
• CRLF Injection Protection: Validates endpoints and headers for HTTP header injection
• Response Size Limits: Configurable max response size (default: 100MB)

Platform Support¶

Windows AF_UNIX support uses the socket2 crate for socket creation and raw Winsock handle conversion to tokio-compatible types. The implementation is isolated in unix_socket_windows.rs and activated via #[cfg(windows)] gates.

Model Aggregation Service¶

Location: src/models/ (modular structure)

Purpose: Aggregate and cache model information from all backends, enrich with metadata, with cache stampede prevention

Module Structure:

src/models/
├── mod.rs             # Public module exports
├── types.rs           # Model, AggregatedModel, ModelList, SingleModelResponse types
├── metrics.rs         # ModelMetrics tracking (includes stampede metrics)
├── cache.rs           # ModelCache with stale-while-revalidate & singleflight
├── config.rs          # ModelAggregationConfig
├── fetcher.rs         # Model fetching from backends
├── handlers.rs        # HTTP handlers for /v1/models and /v1/models/{model} endpoints
├── background_refresh.rs  # Background periodic cache refresh service
├── pattern_matching.rs    # 6-phase model ID lookup pipeline (date / format suffix normalization, HuggingFace repo-prefix stripping, wildcard)
├── utils.rs           # Utility functions (normalize_model_id, etc.)
└── aggregation/       # Core aggregation logic
    ├── mod.rs         # ModelAggregationService implementation
    └── tests.rs       # Unit tests

For the model ID resolution pipeline used during metadata enrichment, see Automatic Quantization and Format Suffix Handling in the configuration guide. For policy guidance on when a new mapping should be added as an explicit YAML alias versus an extension to the peel allowlist or the HuggingFace prefix-strip layer, see Aliases vs. suffix normalization: when to use which. The prefix-strip layer composes with the suffix peel: a single lookup can strip a vendor/repo prefix and then peel an allowlisted suffix token, so unsloth/Qwen3.6-35B-A3B-GGUF routes to qwen3.6-35b-a3b metadata without a hand-registered alias.

Cache Stampede Prevention¶

The model aggregation service implements three strategies to prevent cache stampede (thundering herd problem):

1. Singleflight Pattern: Only one aggregation request runs at a time. Concurrent requests wait for the ongoing aggregation to complete, then share the result.

2. Stale-While-Revalidate: When cache is stale (between soft and hard TTL), return stale data immediately while triggering a background refresh. Clients get fast responses with potentially slightly stale data.

3. Background Periodic Refresh: A background task proactively refreshes the cache before expiration. Requests never block on cache refresh (except cold start).

pub struct ModelAggregationService {
    cache: ModelCache,  // With singleflight lock and background refresh tracking
    config: ModelAggregationConfig,
    fetcher: ModelFetcher,
}

impl ModelAggregationService {
    // Aggregate models with singleflight protection
    pub async fn aggregate_models_with_singleflight(&self, state: &Arc<AppState>)
        -> Result<AggregatedModelsResponse, StatusCode> { /* ... */ }

    // Find backends with stale-while-revalidate support
    pub async fn find_backends_for_model(&self, state: &Arc<AppState>, model_id: &str)
        -> Vec<String> { /* ... */ }

    // Clear cache (used during hot reload)
    pub fn clear_cache(&self) { /* ... */ }
}

Cache TTL Configuration¶

The cache uses a dual-TTL approach:

Metrics¶

New metrics for cache stampede monitoring:

• stale_while_revalidate: Requests that returned stale data during refresh
• coalesced_requests: Requests that waited for ongoing aggregation
• background_refreshes: Background refresh operations initiated
• background_refresh_successes/failures: Background refresh outcomes
• singleflight_lock_acquired: Times the aggregation lock was acquired

Proxy Module¶

Location: src/proxy/ (modular structure)

Purpose: Handle request proxying, backend selection, file resolution, and image generation/editing

Module Structure:

src/proxy/
├── mod.rs             # Public module exports
├── backend.rs         # Backend selection and routing logic
├── request.rs         # Request execution with retry logic
├── files.rs           # File reference resolution in requests
├── image_gen.rs       # Image generation handling (DALL-E, Gemini, GPT Image)
├── image_edit.rs      # Image editing support (/v1/images/edits)
├── image_utils.rs     # Image processing utilities (multipart, validation)
├── handlers.rs        # HTTP handlers for proxy endpoints
├── utils.rs           # Utility functions (error responses, etc.)
└── tests.rs           # Unit tests

Key Responsibilities¶

• Backend Selection: Intelligent routing to available backends
• File Resolution: Resolve file references in chat requests
• Image Generation: Support for OpenAI (DALL-E, GPT Image) and Gemini (Nano Banana) image models
• Image Editing: Image editing and variations endpoints
• Request Retry: Automatic retry with exponential backoff
• Error Handling: Standardized error responses in OpenAI format

Retry Handler¶

Location: src/services/deduplication.rs

Purpose: Implement exponential backoff with jitter and request deduplication

EnhancedRetryHandler composes the core retry engine with a DeduplicationManager; configuration uses the public string-duration RetryConfig (max_attempts, initial_delay, max_delay, backoff_multiplier, jitter, retryable status/error lists, and total timeout). The retry state machine itself lives in src/core/retry/strategy.rs.

pub struct EnhancedRetryHandler {
    pub retry_handler: RetryHandler,
    deduplication: DeduplicationManager,
    enable_deduplication: bool,
}

Circuit Breaker¶

Location: src/core/circuit_breaker/

Purpose: Maintain an independent per-backend three-state circuit-breaker state machine

pub struct CircuitBreaker {
    states: Arc<DashMap<String, BackendCircuitState>>,
    config: CircuitBreakerConfig,
    metrics: Option<CircuitBreakerMetrics>,
}

pub struct CircuitBreakerConfig {
    pub enabled: bool,
    pub failure_threshold: u32,           // Failures before opening (default: 5)
    pub failure_rate_threshold: f64,      // Failure rate threshold (default: 0.5)
    pub minimum_requests: u32,            // Min requests before rate calculation
    pub timeout: Duration,                // Open-state cooldown (default: 60s)
    pub half_open_max_requests: u32,      // Max requests in half-open state
    pub half_open_success_threshold: u32, // Successes needed to close
}

pub enum CircuitState {
    Closed,    // Normal operation - requests pass through
    Open,      // Failing fast - requests rejected immediately
    HalfOpen,  // Testing recovery - limited requests allowed
}

Key Features¶

• Per-backend circuit breakers with independent state
• Atomic operations for lock-free state checking in hot path
• Automatic state transitions based on success/failure patterns
• Sliding window for failure rate calculation
• A metrics collector that can be attached by an embedding application
• Admin endpoints for inspection and manual control

The standard LLM proxy path records backend outcomes and consults the breaker during candidate selection and immediately before dispatch. Open circuits are removed from ordinary inference traffic, half-open probes are capacity-limited, and the normal metrics registry exposes the live breaker counters. Active health filtering, retry, and fallback remain complementary layers.

Container (Dependency Injection)¶

Location: src/core/container.rs

Purpose: Manage service lifecycles and dependencies

pub struct Container {
    services: Arc<RwLock<HashMap<TypeId, Box<dyn Any + Send + Sync>>>>,
    singletons: Arc<RwLock<HashMap<TypeId, Arc<dyn Any + Send + Sync>>>>,
}

impl Container {
    // Register singleton service
    pub async fn register_singleton<T>(&self, instance: Arc<T>) -> CoreResult<()> 
    where T: 'static + Send + Sync { /* ... */ }

    // Resolve service dependency
    pub async fn resolve<T>(&self) -> CoreResult<Arc<T>>
    where T: 'static + Send + Sync { /* ... */ }
}

Data Flow¶

Request Processing Flow¶

sequenceDiagram
    participant Client
    participant HTTPLayer as HTTP Layer
    participant ProxyService as Proxy Service
    participant BackendService as Backend Service
    participant ModelService as Model Service
    participant Backend as LLM Backend

    Client->>HTTPLayer: POST /v1/chat/completions
    HTTPLayer->>HTTPLayer: Apply Middleware (auth, logging, metrics)
    HTTPLayer->>ProxyService: Forward Request

    ProxyService->>ModelService: Get Model Info
    ModelService->>ModelService: Check Cache
    alt Cache Miss
        ModelService->>BackendService: Get Backends for Model
        BackendService->>Backend: Query Models
        Backend-->>BackendService: Model List
        BackendService-->>ModelService: Filtered Backends
        ModelService->>ModelService: Update Cache
    end
    ModelService-->>ProxyService: Model Available on Backends

    ProxyService->>BackendService: Select Healthy Backend
    BackendService->>BackendService: Apply Load Balancing
    BackendService-->>ProxyService: Selected Backend

    ProxyService->>Backend: Forward Request
    Backend-->>ProxyService: Response (streaming or non-streaming)

    ProxyService->>ProxyService: Apply Response Processing
    ProxyService-->>HTTPLayer: Processed Response
    HTTPLayer-->>Client: HTTP Response

Guardrail Layer¶

When guardrails are configured (guardrails.enabled: true), a content-safety layer wraps the backend call on every request path (OpenAI chat completions, the /v1/responses bridge, and native Anthropic Messages). The layer is implemented as GuardrailService in the Services Layer (src/services/guardrail/) and is driven from the proxy handlers through a single gating seam (src/services/guardrail/gate.rs), so the three dispatch paths share the same policy and the same block-response shapes. When guardrails are unconfigured the handlers never enter this seam, so the request flow is unchanged and adds no overhead.

The service lives in AppState behind GuardrailServiceHandle (src/services/guardrail/handle.rs), a lock-free late-settable slot (arc-swap) rather than a startup-frozen Option: request paths load it with a single atomic load (once per request, reused across that request's gates), and a router that started with guardrails disabled can have a service materialized into the slot later, by the guardrail config watcher on a local enabled: true hot-reload or by the control-plane guardrail policy reconciler on an enforceable hub policy. Materialization goes through the same factory path startup uses (guardrail_service_factory in src/server/state.rs), including the backend resolver for backend: provider references. A hub policy that demands enforcement the router cannot execute is rejected whole with the typed code guardrails_unavailable and recorded on the control-plane policy store; request behavior stays on local configuration.

The layer hooks the request lifecycle at three points:

• Input pre-call: before backend dispatch. A Block verdict short-circuits the request without calling the backend; a Transform rewrites the prompt before dispatch.
• Input during-call: concurrently with the backend call via tokio::join!, so a remote classifier's latency overlaps the model latency. A block discards the in-flight response and returns the block instead.
• Output post-call: after the backend returns, over the generated text, before the response is returned or cached. For streaming responses this becomes a rolling-window check (buffer_full / chunked / passthrough).

GuardrailService owns all policy: monitor-versus-enforce mode, the bypass allowlist, per-route provider subsetting, per-category thresholds, allow/deny lists, per-provider timeouts, and the fail-open/fail-closed decision. It aggregates the verdicts of the configured providers (OpenAI Moderation, self-hosted classifier, PII, AWS Bedrock, Azure Content Safety) using a most-severe-wins rule and emits Prometheus metrics and an audit-log entry for every decision.

sequenceDiagram
    participant Client
    participant Gate as Guardrail Gate
    participant Service as GuardrailService
    participant Backend as LLM Backend

    Client->>Gate: request (prompt / messages)
    Gate->>Service: input pre-call
    alt Block (enforce)
        Service-->>Client: block response (content_filter / refusal / error)
    else Allow or Transform
        Service-->>Gate: verdict
        par during-call overlaps dispatch
            Gate->>Backend: forward (possibly transformed) request
            Gate->>Service: input during-call
        end
        Backend-->>Gate: response
        Gate->>Service: output post-call
        alt Block (enforce)
            Service-->>Client: block response
        else Allow or Transform
            Service-->>Client: response (possibly sanitized)
        end
    end

For the operator-facing guide (concepts, provider setup, configuration, threshold tuning, admin controls, and metrics), see Guardrails.

Responses-API Routing for responses_only Models¶

Some OpenAI Pro models (currently gpt-5-pro, gpt-5.2-pro, gpt-5.4-pro, gpt-5.5-pro) are exposed exclusively via /v1/responses and return 404 not_found from /v1/chat/completions. The responses_only: true capability flag in model-metadata.yaml (and the built-in OpenAI registry) marks these models so the router transparently dispatches them to the Responses API surface, regardless of which client surface the request arrived on.

The flag is consulted at the proxy layer right after model lookup, before backend dispatch. The same model metadata pipeline that resolves capability information also resolves responses_only, so the lookup composes with backend-specific overrides, the external metadata file, and built-in defaults in the standard priority order (see Metadata Priority and Alias Resolution).

sequenceDiagram
    participant Client
    participant ChatHandler as POST /v1/chat/completions
    participant AnthropicHandler as POST /anthropic/v1/messages
    participant Lookup as is_responses_only_model()
    participant Converter as Request Converter
    participant Backend as OpenAI / Azure OpenAI

    alt OpenAI surface
        Client->>ChatHandler: chat.completion request
        ChatHandler->>Lookup: (backend, model)
    else Anthropic surface
        Client->>AnthropicHandler: messages request
        AnthropicHandler->>Lookup: (backend, model)
    end

    alt responses_only = true
        alt Backend is OpenAI / Azure OpenAI
            Lookup-->>Converter: Convert to /v1/responses payload
            Converter->>Backend: POST /v1/responses
            Backend-->>Converter: Responses API response (or SSE)
            Converter-->>Client: Translated to original surface shape
        else Backend is not OpenAI / Azure OpenAI
            Lookup-->>Client: 400 invalid_request_error
        end
    else responses_only = false
        Lookup-->>Backend: Standard /v1/chat/completions dispatch
        Backend-->>Client: chat.completion / Anthropic Messages
    end

Key properties:

• The dispatch is transparent on both surfaces: clients keep using /v1/chat/completions or /anthropic/v1/messages and receive responses in the matching shape.
• Only OpenAI and Azure OpenAI backends serve /v1/responses; other backend types are rejected with 400 invalid_request_error before any upstream call.
• The first dispatch per (backend, model) pair logs at info level so operators can confirm Responses-API routing without enabling debug logs.
• Streaming requests are bridged event-by-event: Responses API SSE events are translated into chat.completion.chunk (OpenAI surface) or Anthropic Messages SSE events (Anthropic surface) without buffering the whole response.

For configuration syntax, the current model list, and guidance on marking new models, see Responses-API-only Models in the configuration guide.

Health Check Flow¶

sequenceDiagram
    participant HealthService as Health Service
    participant BackendPool as Backend Pool
    participant Backend as LLM Backend
    participant Cache as Health Cache

    loop Every Interval
        HealthService->>BackendPool: Get All Backends
        BackendPool-->>HealthService: Backend List

        par For Each Backend
            HealthService->>Backend: GET /v1/models (or /health)
            alt Success
                Backend-->>HealthService: 200 OK + Model List
                HealthService->>Cache: Update: consecutive_successes++
                HealthService->>HealthService: Mark Healthy if threshold met
            else Failure
                Backend-->>HealthService: Error/Timeout
                HealthService->>Cache: Update: consecutive_failures++
                HealthService->>HealthService: Mark Unhealthy if threshold met
            end
        end

        HealthService->>BackendPool: Update Backend Health Status
    end

Hot Reload Service¶

Location: src/infrastructure/config/hot_reload.rs, src/infrastructure/config/hot_reload_service.rs

Purpose: Provide runtime configuration updates without server restart

The hot reload system enables zero-downtime configuration changes through automatic file watching and selective component updates.

Key Architecture Components¶

• ConfigManager: File system watching using notify crate, publishes updates via tokio::sync::watch channel
• HotReloadService: Computes configuration differences, classifies changes (immediate/gradual/restart)
• Component Updates: Interior mutability patterns (RwLock) for atomic updates to HealthChecker, CircuitBreaker, RateLimitStore, BackendPool

Backend Pool Hot Reload¶

The BackendPool uses interior mutability (Arc<RwLock<Vec<Arc<Backend>>>>) to support runtime backend additions and removals without service interruption.

Backend States:

• Active: Normal operation, accepts new requests
• Draining: Removed from config, existing requests continue, no new requests
• Removed: Fully cleaned up after all references released

Graceful Draining Process:

1. When a backend is removed from config, it's marked as Draining
2. New requests skip draining backends
3. Existing in-flight requests/streams continue uninterrupted (Arc reference counting)
4. Background cleanup task runs every 10 seconds
5. Backends are removed when: no references remain OR 5-minute timeout exceeded

This ensures zero impact on ongoing connections during configuration changes.

Change Classification¶

• Immediate update: active rate-limit and circuit-breaker policies, global prompts, request parameters, streaming/smart-routing snapshots, selection strategy, prefix routing, the retry policy, per-request timeout budgets, and documented dynamic fields
• Gradual update: backends and health checks
• Requires restart: server/CORS, logging and tracing, the HTTP client's connection and overall timeouts, cache construction, and other startup-built services

Admin API: /admin/config/hot-reload-status for inspecting hot reload capabilities

For detailed hot reload configuration, process flow, and usage examples, see the Hot Reload section in the configuration guide.

Configuration Migration System¶

Location: src/infrastructure/config/{migrator,migration,migrations,fixer,backup}.rs

Purpose: Automatically detect and fix configuration issues, migrate schemas, and ensure configuration validity

The configuration migration system handles configuration evolution and maintenance. It automatically: - Detects and migrates outdated schema versions - Fixes common syntax errors in YAML/TOML files - Validates and corrects configuration values - Creates backups before making changes - Provides dry-run capability for previewing changes

Architecture Components¶

1. Migration Orchestrator (migrator.rs) - Main entry point for migration operations - Coordinates the entire migration workflow - Manages backup creation and restoration - Implements security validations (path traversal, file size limits)

2. Migration Framework (migration.rs) - Defines core types and traits for migrations - Migration trait for implementing version upgrades - ConfigIssue enum for categorizing problems - MigrationResult for tracking changes

3. Schema Migrations (migrations.rs) - Concrete migration implementations (e.g., V1ToV2Migration) - Transforms configuration structure between versions - Example: Converting backend_url to backends array

4. Auto-Correction Engine (fixer.rs) - Detects and fixes common configuration errors - Duration format correction (e.g., "10 seconds" → "10s") - URL validation and protocol addition - Field deprecation handling

5. Backup Manager (backup.rs) - Creates timestamped backups before modifications - Implements resource limits (10MB per file, 100MB total, max 50 backups) - Automatic cleanup of old backups - Preserves file permissions

Migration Workflow¶

graph TD
    A[Read Config File] --> B[Validate Path & Size]
    B --> C[Create Backup]
    C --> D[Parse Configuration]
    D --> E{Parse Success?}
    E -->|No| F[Fix Syntax Errors]
    F --> D
    E -->|Yes| G[Detect Schema Version]
    G --> H{Needs Migration?}
    H -->|Yes| I[Apply Migrations]
    H -->|No| J[Validate Values]
    I --> J
    J --> K{Issues Found?}
    K -->|Yes| L[Apply Auto-Fixes]
    K -->|No| M[Return Config]
    L --> N[Write Updated Config]
    N --> M

Security Features¶

1. Path Traversal Protection: Validates paths to prevent directory traversal attacks
2. File Size Limits: Maximum 10MB configuration files to prevent DoS
3. Format Validation: Only processes .yaml, .yml, and .toml files
4. System Directory Protection: Blocks access to sensitive system paths
5. Test Mode Relaxation: Uses conditional compilation for test-friendly validation

Example Migration: v1.0 to v2.0¶

// V1ToV2Migration implementation
fn migrate(&self, config: &mut Value) -> Result<(), MigrationError> {
    // Convert single backend_url to backends array
    if let Some(backend_url) = config.get("backend_url") {
        let mut backends = Vec::new();
        let mut backend = Map::new();
        backend.insert("url".to_string(), backend_url.clone());

        // Move models to backend
        if let Some(model) = config.get("model") {
            backend.insert("models".to_string(), 
                Value::Sequence(vec![model.clone()]));
        }

        backends.push(Value::Mapping(backend));
        config["backends"] = Value::Sequence(backends);

        // Remove old fields
        config.remove("backend_url");
        config.remove("model");
    }
    Ok(())
}

Configuration Loading Flow¶

graph TD
    A[Application Start] --> B[Config Manager Init]
    B --> C{Config File Specified?}
    C -->|Yes| D[Load Specified File]
    C -->|No| E[Search Standard Locations]

    E --> F{Config File Found?}
    F -->|Yes| G[Load Config File]
    F -->|No| H[Use CLI Args + Env Vars + Defaults]

    D --> I[Parse YAML]
    G --> I
    H --> J[Create Config from Args]

    I --> K[Apply Environment Variable Overrides]
    J --> K

    K --> L[Apply CLI Argument Overrides]
    L --> M[Validate Configuration]
    M --> N{Valid?}
    N -->|Yes| O[Return Config]
    N -->|No| P[Exit with Error]

    O --> Q[Start File Watcher for Hot Reload]
    Q --> R[Application Running]

    Q --> S[Config File Changed]
    S --> T[Reload and Validate]
    T --> U{Valid?}
    U -->|Yes| V[Apply New Config]
    U -->|No| W[Log Error, Keep Old Config]

    V --> R
    W --> R

Dependency Injection¶

Service Registration¶

Services are registered in the container during application startup:

// In main.rs
async fn setup_services(config: Config) -> Result<ServiceRegistry, Error> {
    let container = Arc::new(Container::new());

    // Register infrastructure services
    container.register_singleton(Arc::new(
        HttpClient::new(&config.http_client)?
    )).await?;

    container.register_singleton(Arc::new(
        BackendManager::new(&config.backends)?
    )).await?;

    // Register core services
    container.register_singleton(Arc::new(
        BackendServiceImpl::new(container.clone())
    )).await?;

    container.register_singleton(Arc::new(
        ModelServiceImpl::new(container.clone())
    )).await?;

    // Create service registry
    let registry = ServiceRegistry::new(container);
    registry.initialize().await?;

    Ok(registry)
}

Service Dependencies¶

Services declare their dependencies through constructor injection:

pub struct ProxyServiceImpl {
    backend_service: Arc<dyn BackendService>,
    model_service: Arc<dyn ModelService>,
    retry_handler: Arc<dyn RetryHandler>,
    http_client: Arc<HttpClient>,
}

impl ProxyServiceImpl {
    pub fn new(container: Arc<Container>) -> CoreResult<Self> {
        Ok(Self {
            backend_service: container.resolve()?,
            model_service: container.resolve()?,
            retry_handler: container.resolve()?,
            http_client: container.resolve()?,
        })
    }
}

Benefits¶

• Testability: Services can be mocked for unit testing
• Flexibility: Implementations can be swapped without code changes
• Lifecycle Management: Container manages service initialization and cleanup
• Circular Dependency Detection: Container prevents circular dependencies

Error Handling Strategy¶

The router implements an error handling strategy with typed errors, automatic recovery, and OpenAI-compatible error responses.

Error Type Hierarchy¶

• CoreError: Domain-level errors (validation, service failures, timeouts, configuration)
• RouterError: Application-level errors combining Core, HTTP, Backend, and Model errors
• HttpError: HTTP-specific errors (400 BadRequest, 401 Unauthorized, 404 NotFound, 500 InternalServerError, etc.)

Error Handling Principles¶

1. Fail Fast: Validate inputs early with clear error messages
2. Error Context: Include relevant context (field names, operation details)
3. Retryable Classification: Distinguish between retryable (timeout, 503) and non-retryable (400, 401) errors
4. User-Friendly Responses: Convert internal errors to OpenAI-compatible error format
5. Structured Logging: Log errors with appropriate severity and context

Error Recovery Mechanisms¶

• Retry with Exponential Backoff: Automatically retry transient failures
• Model Fallback: Route to alternative models when primary unavailable (see Model Fallback System)
• Graceful Degradation: Continue with reduced functionality when components fail

For detailed error handling, recovery strategies, monitoring, and troubleshooting, see error-handling.md.

Extension Points¶

Backend Type Architecture¶

The router supports multiple backend types with different API formats. Each backend type handles request/response transformation automatically.

Supported Backend Types¶

Anthropic Backend Architecture¶

The Anthropic backend provides native support for Claude models with automatic format translation:

Key Transformations¶

Request Format Differences:

Extended Thinking Support:

// OpenAI reasoning_effort → Anthropic thinking
{
  "reasoning_effort": "high"  // OpenAI format
}
// Transforms to:
{
  "thinking": {
    "type": "enabled",
    "budget_tokens": 32768   // Mapped from effort level
  }
}

Tool Calling Transformation¶

The router provides automatic transformation of OpenAI-format tool definitions to backend-native formats, enabling cross-provider tool calling support.

Location: src/core/tool_calling/

Module Structure:

src/core/tool_calling/
├── mod.rs           # Module exports
├── definitions.rs   # Type definitions (ToolDefinition, FunctionDefinition, etc.)
├── transform.rs     # Transformation functions for each backend
├── tool_choice.rs   # Tool choice transformation for each backend
├── response.rs      # Tool call response extraction from backends
├── streaming.rs     # Streaming tool call transformation
└── messages.rs      # Multi-turn conversation message transformation

Supported Backends¶

OpenAI Tool Format (Input)¶

{
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": { "type": "string" }
          },
          "required": ["location"]
        }
      }
    }
  ]
}

Anthropic Transformation¶

// transform_tools_to_anthropic()
// Input:  OpenAI format with `function.parameters`
// Output: Anthropic format with `input_schema`
{
  "tools": [
    {
      "name": "get_weather",
      "description": "Get current weather for a location",
      "input_schema": {
        "type": "object",
        "properties": {
          "location": { "type": "string" }
        },
        "required": ["location"]
      }
    }
  ]
}

Gemini Transformation¶

// transform_tools_to_gemini()
// Input:  OpenAI format
// Output: Gemini nested functionDeclarations format
{
  "tools": [
    {
      "functionDeclarations": [
        {
          "name": "get_weather",
          "description": "Get current weather for a location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": { "type": "string" }
            },
            "required": ["location"]
          }
        }
      ]
    }
  ]
}

Validation Rules¶

• Tool Name: Must match pattern ^[a-zA-Z0-9_-]+$ (alphanumeric, underscore, hyphen only)
• Empty Name: Rejected with validation error
• Missing Fields: Logged as warnings, tool skipped (no silent data loss)
• Type Field: Must be "function" for standard tool definitions

Integration with Fallback System¶

The tool calling transformation integrates with the model fallback system via ParameterTranslator:

// In src/core/fallback/translation.rs
impl ParameterTranslator {
    pub fn translate_tools(
        &self,
        tools: &Value,
        from_backend_type: BackendType,
        to_backend_type: BackendType,
    ) -> CoreResult<Value> {
        match (from_backend_type, to_backend_type) {
            (_, BackendType::Anthropic) => transform_tools_to_anthropic(tools),
            (_, BackendType::Gemini) => transform_tools_to_gemini(tools),
            (_, BackendType::LlamaCpp) => transform_tools_for_llamacpp(tools),
            _ => Ok(tools.clone()),
        }
    }
}

This enables fallback from one provider to another while preserving tool definitions.

Tool Choice Transformation¶

In addition to tool definitions, the router also transforms the tool_choice parameter which controls model tool-calling behavior.

Supported Values:

Edge Cases:

• Anthropic "none" workaround: Anthropic API does not support tool_choice=none. When this value is detected, the router removes tools and tool_choice entirely from the request.
• llama.cpp: Preserves parallel_tool_calls parameter for parallel function calling support.

Implementation:

// In src/core/tool_calling/tool_choice.rs
pub enum ToolChoiceValue {
    Auto,
    None,
    Required,
    Function(String),
}

impl ToolChoiceValue {
    pub fn from_openai(value: &Value) -> CoreResult<Self>;
    pub fn to_anthropic(&self) -> Option<Value>;  // Returns None for "none"
    pub fn to_gemini(&self) -> Value;
}

Streaming Tool Call Transformation¶

When streaming is enabled, tool calls are returned incrementally as delta events rather than complete objects. Different providers use distinct streaming formats, requiring real-time transformation to maintain OpenAI API compatibility.

Location: src/core/tool_calling/streaming.rs

Key Components:

Streaming Format Comparison:

State Machine (Anthropic):

IDLE → message_start → READY
READY → content_block_start (tool_use) → TOOL_STARTED
TOOL_STARTED → content_block_delta → ACCUMULATING
ACCUMULATING → content_block_delta → ACCUMULATING (loop)
ACCUMULATING → content_block_stop → TOOL_COMPLETE
TOOL_COMPLETE → message_delta → FINISHED

Safety Limits:

To prevent memory exhaustion from malformed streams:

• MAX_TOOL_CALLS = 64: Maximum parallel tool calls per message
• MAX_ARGUMENTS_SIZE = 1MB: Maximum accumulated arguments per tool call

Example Transformation (Anthropic → OpenAI):

// Anthropic input: content_block_start
{"type": "content_block_start", "index": 0,
 "content_block": {"type": "tool_use", "id": "toolu_123", "name": "get_weather"}}

// OpenAI output: delta chunk
{"id": "chatcmpl-...", "choices": [{"delta": {"tool_calls": [
  {"index": 0, "id": "toolu_123", "type": "function",
   "function": {"name": "get_weather", "arguments": ""}}
]}}]}

Finish Reason Mapping:

Multi-Turn Conversation Message Transformation¶

When using tools in multi-turn conversations, the message history contains tool calls from the assistant and tool results from the client. These messages require format transformation when routing to different backends.

Location: src/core/tool_calling/messages.rs

Key Functions:

Message Format Differences:

OpenAI Format (Input):

{
  "messages": [
    {"role": "user", "content": "What's the weather in NYC?"},
    {
      "role": "assistant",
      "tool_calls": [{
        "id": "call_abc123",
        "type": "function",
        "function": {"name": "get_weather", "arguments": "{\"location\": \"NYC\"}"}
      }]
    },
    {
      "role": "tool",
      "tool_call_id": "call_abc123",
      "content": "{\"temperature\": 72, \"condition\": \"sunny\"}"
    }
  ]
}

Anthropic Transformation:

{
  "messages": [
    {"role": "user", "content": "What's the weather in NYC?"},
    {
      "role": "assistant",
      "content": [{
        "type": "tool_use",
        "id": "call_abc123",
        "name": "get_weather",
        "input": {"location": "NYC"}
      }]
    },
    {
      "role": "user",
      "content": [{
        "type": "tool_result",
        "tool_use_id": "call_abc123",
        "content": "{\"temperature\": 72, \"condition\": \"sunny\"}"
      }]
    }
  ]
}

Gemini Transformation:

{
  "contents": [
    {"role": "user", "parts": [{"text": "What's the weather in NYC?"}]},
    {
      "role": "model",
      "parts": [{
        "functionCall": {"name": "get_weather", "args": {"location": "NYC"}}
      }]
    },
    {
      "role": "function",
      "parts": [{
        "functionResponse": {
          "name": "get_weather",
          "response": {"temperature": 72, "condition": "sunny"}
        }
      }]
    }
  ]
}

Multiple Tool Results:

When multiple tools are called in parallel, consecutive tool result messages are combined for Anthropic into a single user message:

// Multiple OpenAI tool results:
{"role": "tool", "tool_call_id": "call_1", "content": "72F, sunny"}
{"role": "tool", "tool_call_id": "call_2", "content": "3:00 PM EST"}

// Combined Anthropic format:
{
  "role": "user",
  "content": [
    {"type": "tool_result", "tool_use_id": "call_1", "content": "72F, sunny"},
    {"type": "tool_result", "tool_use_id": "call_2", "content": "3:00 PM EST"}
  ]
}

Error Handling:

• Missing tool_call_id: Returns validation error
• Malformed tool calls: Logged as warnings, skipped without failing entire request
• Unknown function name for Gemini: Falls back to "unknown" with warning log
• Malformed JSON arguments: Falls back to empty object {}
• is_error indicator: Preserved when transforming to Anthropic format

Gemini 3 thoughtSignature Support¶

Gemini 3 function calls use a thoughtSignature field that must be preserved and passed back in multi-turn conversations. The router handles this through the transformation pipeline.

What is thoughtSignature?

When Gemini 3+ models return function calls, they include an encrypted thoughtSignature field that encapsulates the model's internal reasoning context. This signature must be included when sending the function result back to continue the conversation correctly.

Response Extraction (Gemini -> Client):

The router extracts thoughtSignature from Gemini responses and includes it in the OpenAI-compatible format:

// Gemini native response (with thoughtSignature)
{
  "candidates": [{
    "content": {
      "parts": [{
        "functionCall": {"name": "get_weather", "args": {"location": "NYC"}},
        "thoughtSignature": "encrypted_signature_abc123"
      }]
    }
  }]
}

// Router's OpenAI-compatible response
{
  "choices": [{
    "message": {
      "tool_calls": [{
        "id": "call_xyz789",
        "type": "function",
        "function": {"name": "get_weather", "arguments": "{\"location\":\"NYC\"}"},
        "extra_content": {
          "google": {
            "thought_signature": "encrypted_signature_abc123"
          }
        }
      }]
    }
  }]
}

Request Injection (Client -> Gemini):

When clients send tool results back, the router extracts thought_signature from extra_content.google and injects it as thoughtSignature in the Gemini request:

// Client's OpenAI-format request
{
  "messages": [{
    "role": "assistant",
    "tool_calls": [{
      "id": "call_xyz789",
      "function": {"name": "get_weather", "arguments": "{}"},
      "extra_content": {
        "google": {"thought_signature": "encrypted_signature_abc123"}
      }
    }]
  }]
}

// Transformed Gemini request
{
  "contents": [{
    "role": "model",
    "parts": [{
      "functionCall": {"name": "get_weather", "args": {}},
      "thoughtSignature": "encrypted_signature_abc123"
    }]
  }]
}

Streaming Support:

The streaming transformer also handles thoughtSignature extraction, including it in tool call delta chunks.

Compatibility Behavior:

• Gemini 2.x models do not return thoughtSignature - the router gracefully handles this
• Clients that don't preserve extra_content.google.thought_signature can still make function calls (but may have degraded conversation continuity on Gemini 3+)
• The extra_content field is ignored by non-Gemini backends

Implementation Locations:

Adding New Backend Types¶

1. Implement Backend Trait:

   // In src/infrastructure/backends/custom_backend.rs
   pub struct CustomBackend {
       client: Arc<HttpClient>,
       config: CustomBackendConfig,
   }
   
   #[async_trait]
   impl BackendTrait for CustomBackend {
       async fn health_check(&self) -> CoreResult<()> { /* ... */ }
       async fn list_models(&self) -> CoreResult<Vec<Model>> { /* ... */ }
       async fn chat_completion(&self, request: ChatRequest) -> CoreResult<Response> { /* ... */ }
   }
   

2. Register in Backend Factory:

   // In src/infrastructure/backends/mod.rs
   pub fn create_backend(backend_type: &str, config: &BackendConfig) -> CoreResult<Box<dyn BackendTrait>> {
       match backend_type {
           "openai" => Ok(Box::new(OpenAIBackend::new(config)?)),
           "vllm" => Ok(Box::new(VLLMBackend::new(config)?)),
           "custom" => Ok(Box::new(CustomBackend::new(config)?)), // New backend
           _ => Err(CoreError::ValidationFailed {
               message: format!("Unknown backend type: {}", backend_type),
               field: Some("backend_type".to_string()),
           }),
       }
   }
   

Adding New Middleware¶

1. Implement Middleware Trait:

   // In src/http/middleware/custom_middleware.rs
   pub struct CustomMiddleware {
       config: CustomConfig,
   }
   
   impl<S> tower::Layer<S> for CustomMiddleware {
       type Service = CustomMiddlewareService<S>;
   
       fn layer(&self, inner: S) -> Self::Service {
           CustomMiddlewareService { inner, config: self.config.clone() }
       }
   }
   

2. Register in HTTP Router:

   // In src/main.rs
   let app = Router::new()
       .route("/v1/models", get(list_models))
       .layer(CustomMiddleware::new(config.custom))
       .layer(LoggingMiddleware::new())
       .with_state(state);
   

Adding New Cache Types¶

1. Implement the CacheStore trait:

   // In src/infrastructure/cache/your_store.rs
   pub struct YourCacheStore { /* ... */ }
   
   #[async_trait]
   impl CacheStore for YourCacheStore {
       async fn get(&self, key: &str) -> CacheStoreResult<Option<Vec<u8>>> { /* ... */ }
       async fn set(&self, key: &str, value: &[u8], ttl: Duration) -> CacheStoreResult<()> { /* ... */ }
       async fn delete(&self, key: &str) -> CacheStoreResult<()> { /* ... */ }
       async fn clear(&self) -> CacheStoreResult<()> { /* ... */ }
       async fn stats(&self) -> CacheStoreStats { /* ... */ }
   }
   

The RedisCacheStore (src/infrastructure/cache/redis_store.rs) is a reference implementation, gated behind the redis-cache Cargo feature.

1. Inject into ResponseCacheStore:

   let store: Arc<dyn CacheStore> = Arc::new(YourCacheStore::new(/* ... */));
   let cache = ResponseCacheStore::with_store(store, capacity, ttl, max_response_size, max_stream_buffer_size);
   

Adding New Load Balancing Strategies¶

// In src/services/load_balancer.rs
pub enum LoadBalancingStrategy {
    RoundRobin,
    WeightedRoundRobin,
    LeastConnections,  // New strategy
    Random,
}

impl LoadBalancingStrategy {
    pub fn select_backend(&self, backends: &[Backend]) -> Option<&Backend> {
        match self {
            Self::RoundRobin => /* ... */,
            Self::WeightedRoundRobin => /* ... */,
            Self::LeastConnections => self.select_least_connections(backends),
            Self::Random => /* ... */,
        }
    }
}

Design Decisions¶

Why 4-Layer Architecture?¶

Decision: Use a 4-layer architecture (HTTP → Services → Infrastructure → Core)

Rationale¶

• Clear Separation: Each layer has distinct responsibilities
• Testability: Layers can be tested independently
• Maintainability: Changes in one layer don't affect others
• Flexibility: Easy to swap implementations (e.g., different cache backends)

Trade-offs¶

• ✅ Pros: Clean, maintainable, testable, extensible
• ❌ Cons: More complexity, slight performance overhead
• Verdict: Benefits outweigh costs for a production system

Why Dependency Injection?¶

Decision: Use a custom DI container instead of compile-time injection

Rationale¶

• Runtime Flexibility: Can swap implementations based on configuration
• Service Lifecycle: Centralized management of service initialization/cleanup
• Testing: Easy to inject mocks and test doubles

Alternatives Considered¶

• Manual dependency passing: Too verbose and error-prone
• Compile-time DI (generics): Less flexible, harder to configure

Why Arc> for Shared State?¶

Decision: Use Arc<RwLock<T>> for shared mutable state

Rationale¶

• Reader-Writer Semantics: Multiple readers, exclusive writers
• Performance: Better than Arc<Mutex<T>> for read-heavy workloads
• Safety: Prevents data races at compile time

Alternatives Considered¶

• Arc<Mutex<T>>: Simpler but worse performance for reads
• Channels: Too complex for simple shared state
• Atomic types: Not suitable for complex data structures

Why async/await Throughout?¶

Decision: Use async/await for all I/O operations

Rationale¶

• Performance: Non-blocking I/O allows high concurrency
• Resource Efficiency: Lower memory usage than thread-per-request
• Ecosystem: Rust async ecosystem (Tokio, reqwest, axum) is mature

Trade-offs¶

• ✅ Pros: High performance, low resource usage, good ecosystem
• ❌ Cons: Complexity, learning curve, debugging challenges
• Verdict: Essential for high-performance network services

Why Configuration Hot-Reload?¶

Decision: Support configuration hot-reload using file watching

Rationale¶

• Zero Downtime: Update configuration without restarting
• Operations Friendly: Easy to adjust settings in production
• Development: Faster iteration during development

Implementation¶

• File system watcher detects changes
• Validate new configuration before applying
• Atomic updates to avoid inconsistent state
• Fallback to previous config on validation errors

Performance Considerations¶

Memory Management¶

1. Connection Pooling: Reuse HTTP connections to reduce allocation overhead
2. Smart Caching: LRU eviction prevents unbounded memory growth
3. Arc Cloning: Cheap reference counting instead of deep cloning
4. Streaming: Process responses in chunks to avoid loading large responses into memory

Concurrency¶

1. RwLock for Read-Heavy Workloads: Multiple concurrent readers for backend pool and model cache
2. Lock-Free Where Possible: Use atomics for counters and simple state
3. Async Task Spawning: Background tasks for health checks and cache updates
4. Bounded Channels: Prevent unbounded queuing of tasks

I/O Optimization¶

1. Connection Keep-Alive: TCP connections stay open for reuse
2. Streaming Responses: Forward SSE chunks without buffering
3. Timeouts: Prevent hanging on slow backends
4. Retry with Backoff: Avoid overwhelming failing backends

Memory Layout¶

// Optimized data structures for cache efficiency
pub struct Backend {
    pub name: String,          // Inline string for small names
    pub url: Arc<str>,         // Shared string for URL
    pub weight: u32,           // Compact integer
    pub is_healthy: AtomicBool, // Lock-free health status
}

// Cache-friendly model storage
pub struct ModelCache {
    models: HashMap<String, Arc<ModelInfo>>, // Shared model info
    last_updated: AtomicU64,                 // Lock-free timestamp
    ttl: Duration,
}

Benchmarking Results¶

Based on our benchmarks (see benches/performance_benchmarks.rs):

• Request Latency: < 5ms overhead for routing decisions
• Memory Usage: ~50MB base memory, scales linearly with backends
• Throughput: 1000+ requests/second on modest hardware
• Connection Efficiency: 100+ concurrent connections per backend with minimal memory overhead

Rate Limiting¶

The router implements rate limiting to protect against abuse and ensure fair resource allocation across clients.

Key Features: - Dual-window approach: sustained limit (100 req/min) + burst protection (20 req/5s) - Client identification by API key (preferred) or IP address (fallback) - Per-client isolation with automatic cache cleanup - DoS prevention with short TTL for empty responses

Rate Limit V2 Architecture¶

The enhanced rate limiting system (rate_limit_v2/) provides a modular, high-performance implementation:

Module Structure¶

src/http/middleware/rate_limit_v2/
├── mod.rs              # Public API and module exports
├── middleware.rs       # Axum middleware integration
├── store.rs            # Rate limit storage and client tracking
└── token_bucket.rs     # Token bucket algorithm implementation

Components¶

1. Token Bucket Algorithm (token_bucket.rs)
2. Configurable bucket capacity and refill rate
3. Atomic operations for lock-free token consumption
4. Automatic token replenishment based on elapsed time

5. Separate buckets for sustained and burst limits

6. Rate Limit Store (store.rs)

7. Per-client state tracking with DashMap for concurrent access
8. Automatic cleanup of expired client entries
9. Configurable TTL for inactive clients (default: 1 hour)

10. Memory-efficient with bounded storage

11. Middleware Integration (middleware.rs)

12. Extracts client identifier (API key → IP address fallback)
13. Checks both sustained and burst limits before processing
14. Returns HTTP 429 (Too Many Requests) with Retry-After header
15. Prometheus metrics for monitoring rate limit hits

Configuration Example¶

rate_limiting:
  enabled: true
  sustained:
    max_requests: 100
    window_seconds: 60
  burst:
    max_requests: 20
    window_seconds: 5
  cleanup_interval_seconds: 300

Decision Flow¶

Request arrives
    ↓
Extract client ID (API key or IP)
    ↓
Check sustained limit (100 req/min)
    ↓ OK
Check burst limit (20 req/5s)
    ↓ OK
Process request

For detailed configuration information, see the Rate Limiting section in the configuration guide.

Model Fallback System¶

The router implements a configurable model fallback system that automatically routes requests to alternative models when the primary model is unavailable.

Key Features: - Automatic fallback chain execution (e.g., gpt-4o → gpt-4-turbo → gpt-3.5-turbo) - Cross-provider fallback support with parameter translation - Triggering from configured HTTP errors, timeouts, connection failures, model lookup failures, and backend health - Prometheus metrics for monitoring fallback usage

For detailed configuration and implementation, see error-handling.md section on model fallback.

Circuit Breaker¶

The router contains a per-backend circuit-breaker state machine with Admin inspection and force controls. The ordinary LLM proxy path records request outcomes in it and consults it during selection: an open backend circuit is excluded so traffic routes around it, and when every compatible circuit is open the router returns the standard service-unavailable error and participates in configured fallback. It complements active health checks, retry, and model fallback on that path.

Three-State Machine:

Key Features: - Per-backend isolation with independent state - Lock-free atomic operations for minimal hot-path overhead - Admin endpoints for manual control (/admin/circuit/*) - An optional metrics collector for embedding integrations; it is not registered by the standard server

For detailed configuration and implementation, see error-handling.md section on circuit breaker.

File Storage¶

The router provides OpenAI Files API compatible file storage with persistent metadata.

Key Features: - Persistent metadata storage with sidecar JSON files - Automatic recovery on server restart - Orphan file detection and cleanup - Pluggable backends (memory/persistent)

For detailed architecture and implementation, see File Storage Guide.

Image Generation Architecture¶

The router provides a unified interface for image generation across multiple backends (OpenAI GPT Image, DALL-E, and Google Gemini/Nano Banana) with automatic parameter translation.

Multi-Backend Image Generation¶

OpenAI → Gemini Parameter Conversion¶

When using Nano Banana (Gemini) models, OpenAI-style parameters are automatically converted to Gemini's native format:

Size to Aspect Ratio Mapping¶

Request Transformation¶

OpenAI Format (Input):

{
  "model": "nano-banana",
  "prompt": "A serene Japanese garden",
  "size": "1536x1024",
  "n": 1
}

Gemini Format (Converted):

{
  "contents": [
    {
      "parts": [{"text": "A serene Japanese garden"}]
    }
  ],
  "generationConfig": {
    "imageConfig": {
      "aspectRatio": "3:2",
      "imageSize": "1K"
    }
  }
}

Conversion Implementation¶

The conversion is handled by src/infrastructure/backends/gemini/image_generation.rs:

pub fn convert_openai_to_gemini(request: &OpenAIImageRequest)
    -> CoreResult<(String, GeminiImageRequest)>
{
    // 1. Map model name
    let gemini_model = map_model_to_gemini(&request.model);

    // 2. Parse size to aspect ratio and size category
    let parsed_size = parse_openai_size(&request.size, &request.model)?;

    // 3. Build Gemini request with imageConfig
    let gemini_request = GeminiImageRequest {
        contents: vec![GeminiContent { parts: vec![...] }],
        generation_config: Some(GeminiGenerationConfig {
            image_config: Some(GeminiImageConfig {
                aspect_ratio: Some(parsed_size.aspect_ratio.to_gemini_string()),
                image_size: Some(parsed_size.size_category.to_gemini_image_size()),
            }),
        }),
    };

    Ok((gemini_model, gemini_request))
}

Streaming Image Generation (SSE)¶

For GPT Image models, the router supports true SSE passthrough for streaming image generation:

┌─────────────┐                ┌─────────────┐                ┌─────────────┐
│   Client    │────stream:true─▶│   Router    │────stream:true─▶│   OpenAI    │
│             │                │             │                │             │
│             │◀───SSE events──│  Passthrough│◀───SSE events──│             │
└─────────────┘                └─────────────┘                └─────────────┘

SSE Event Types:

Implementation (src/proxy/image_gen.rs):

async fn handle_streaming_image_generation(...) -> Result<Response, StatusCode> {
    // 1. Keep stream: true in backend request
    // 2. Make streaming request via bytes_stream()
    // 3. Forward SSE events through tokio channel

    let (tx, rx) = tokio::sync::mpsc::unbounded_channel();

    tokio::spawn(async move {
        let mut stream = backend_response.bytes_stream();
        while let Some(chunk) = stream.next().await {
            // Parse SSE format (event:/data: lines)
            // Forward events to client
            for line in chunk_str.lines() {
                if let Some(event_type) = line.strip_prefix("event:") { ... }
                if let Some(data) = line.strip_prefix("data:") {
                    let event = Event::default().event(event_type).data(data);
                    tx.send(Ok(event));
                }
            }
        }
    });

    Ok(Sse::new(UnboundedReceiverStream::new(rx)).into_response())
}

GPT Image Model Features¶

The router supports enhanced parameters for GPT Image models (gpt-image-1, gpt-image-1.5, gpt-image-1-mini):

Model Support Matrix¶

Image Edit and Variations¶

The router provides OpenAI-compatible image editing and variations endpoints through /v1/images/edits and /v1/images/variations.

Image Editing (/v1/images/edits)¶

Endpoint: POST /v1/images/edits

Allows editing an existing image with a text prompt and optional mask. Supported by GPT Image models and DALL-E 2.

Request Format (multipart/form-data):

image: <file>           # Original image (PNG, required)
prompt: <string>        # Edit instructions (required)
mask: <file>            # Optional mask image (PNG)
model: <string>         # Model name (e.g., "gpt-image-1", "dall-e-2")
n: <integer>            # Number of images (default: 1)
size: <string>          # Output size (e.g., "1024x1024")
response_format: <string> # "url" or "b64_json"

Implementation (src/proxy/image_edit.rs): - Multipart form parsing for image and mask files - Image validation (format, size, aspect ratio) - Model-specific parameter transformation - Proper error handling for invalid inputs

Supported Features¶

• Transparent PNG mask support for targeted editing
• Multiple image generation (n parameter)
• Flexible output sizes
• Both URL and base64 response formats

Image Variations (/v1/images/variations)¶

Endpoint: POST /v1/images/variations

Creates variations of a given image. Supported by DALL-E 2 only.

Request Format (multipart/form-data):

image: <file>           # Source image (PNG, required)
model: <string>         # Model name (default: "dall-e-2")
n: <integer>            # Number of variations (default: 1, max: 10)
size: <string>          # Output size ("256x256", "512x512", "1024x1024")
response_format: <string> # "url" or "b64_json"

Implementation (src/proxy/image_edit.rs): - Image file validation and preprocessing - DALL-E 2-specific routing - Error handling for unsupported models - Consistent response formatting

Key Features¶

• Generate multiple variations in a single request
• Automatic image format validation
• Standard OpenAI response format compatibility

Image Utilities Module¶

The image_utils.rs module provides shared utilities for image processing:

Functions¶

• validate_image_format(): Validates PNG/JPEG format and dimensions
• parse_multipart_image_request(): Extracts images from multipart forms
• check_image_dimensions(): Validates size constraints
• format_image_error_response(): Standardized error responses

Validation Rules¶

• Maximum file size: 4MB (configurable)
• Supported formats: PNG (required for edits/variations), JPEG (generation only)
• Aspect ratio constraints per model
• Transparent PNG requirement for masks

Control-Plane Agent (Continuum Hub)¶

The control-plane agent is an optional, outbound-only task that connects a router to a Continuum Hub control plane for fleet management: enrollment, liveness, usage metering, (opt-in) policy sync with local enforcement, (opt-in) fleet request-parameter configuration, (opt-in) provider batch dispatch with lifecycle reporting, and hub-defined synthetic provider/model probes with a router-local monthly budget ceiling. The hub is never in the request path. Routers remain the data plane and connect out to the hub; no inbound hub traffic reaches the router.

The agent is gated behind the control-plane Cargo feature, which stays out of the crate's default/full feature set. Official release binaries are built with the feature enabled in the Release workflow, so for the GitHub Release archives, the Docker images, and the .deb packages built from them the opt-in is the control_plane.enabled config switch, which defaults to false. A source build without the feature carries no extra dependency and no runtime cost. The control_plane config section is always parsed, so config validate accepts it in any build; a feature-less build that sets control_plane.enabled: true logs a warning at startup and does not start the agent.

What the agent does¶

• Enroll: On first start, it exchanges a single-use enrollment_token for a per-router credential (tenant_id, router_id, router_credential). On a 409 name conflict it retries with a fresh name.
• Persist credential: The credential is written to state_file (default ./data/control-plane/state.json, 0600 on Unix) and reused on restart, so the single-use token is not spent again. It is never logged. If the hub later rejects the stored credential, the agent logs an error and backs off; it does not auto-re-enroll in v0 (removing the state file forces a fresh enrollment).
• Heartbeat: On the configured interval it sends a heartbeat with a router inventory: router version, configured backends with per-backend health, the models served, an aggregate health summary, token-supply and saturation telemetry, and cumulative guardrail counters. When policy sync is active, each beat also echoes the applied cursor plus the typed policy capabilities this router actually enforces, last-known-good cursor/digest, and optional value-free rejection evidence so the Hub can distinguish applied, pending, unsupported, and rejected revisions.
• Fleet configuration: When control_plane.config_sync.enabled is set, enrollment and every inventory-carrying heartbeat include a full masked request_params snapshot. The agent polls, fetches, validates, and atomically applies revision-bound public parameter policy without admitting inbound Hub traffic.
• Usage push: It buffers one metadata-only usage record per proxied request (built from the existing token-usage path) and pushes them in bounded batches on the configured cadence.
• Reconnect with backoff: Enrollment, heartbeat, and usage-push failures use exponential backoff with full jitter (base 1s, cap 60s), reset on success.

Privacy invariant: metering metadata only¶

Usage records carry only metering metadata: token counts, model, provider, latency, cache and batch flags, ids, and timestamps. Prompt and completion bodies never leave the router. The wire type (continuum_protocol::UsageRecord) has a closed, fully typed field set, so there is no field that could carry request or response content.

Idempotent batching¶

Each batch is formed once with a stable batch_id and stable per-record idempotency_keys. On a push failure the same batch is retried after backoff; the hub deduplicates on idempotency_key, so a duplicate delivery after a reconnect does not double-count. On any 200 ack the batch is dropped from the buffer. The buffer is bounded (drop-oldest with a warning and a counter), so a prolonged hub outage cannot grow memory without bound.

Cadence and policy¶

The hub returns cadence settings (heartbeat interval, usage-push interval, batch size) in the enrollment response, and those govern the agent. A per-field override in the control_plane config section wins over the hub value for that one setting. Policy sync and enforcement are opt-in via control_plane.policy.enabled; without that block the agent is metering-only and does not call the policy endpoints.

Fleet request-parameter configuration¶

This subsystem is a separate opt-in through control_plane.config_sync.enabled; enabling the agent or policy enforcement alone does not activate it. Its remotely manageable surface is deliberately closed to the public, typed request_params policy. It does not accept arbitrary dotted keys, environment substitutions, provider credentials, router credentials, TLS material, prompts, or completions. An operator may pin the entire section with request_params_immutable: true; otherwise every leaf explicitly present in the local request_params configuration remains a local pin and takes precedence over Hub-owned leaves.

Capability and observation. When the section is mutable, the Router advertises structured section schema version 1, path encoding version 1, hot reload and dry-run support, and hard bounds: 1,820 leaves, six path segments, 256 UTF-8 bytes per segment, 1 MiB encoded content, and 64 exact model scopes. Its inventory carries a complete masked snapshot on enrollment and every inventory-bearing heartbeat. The snapshot contains revision/section/leaf digests, exact path segments, source (hub, local_file, or local_pin), public sensitivity, and reload class, but has no value field. Model identifiers containing /, :, ., or Unicode remain one exact path segment rather than being split or normalized. With request_params_immutable: true, the Router instead advertises only a generic immutable section and its aggregate masked digest; it omits the structured delivery capability and per-leaf entries so Hub cannot assign content the Router has declared immutable.

Outbound transaction. The agent polls POST /api/agent/v1/config/sync; a pending assignment contains identities and masked metadata only. It then fetches exact public content from POST /api/agent/v1/config/content, echoing the authenticated router id, apply id, desired revision/digest, and schema version. Before interpreting a field, the Router requires every identity to agree and recomputes the Hub-compatible canonical SHA-256 digest over the sorted, length-prefixed typed paths and scalar values. It accepts only the registered defaults, overrides, and limits.min|max leaves at global or exact-model scope, with the same numeric and model-key validation as local configuration.

Atomic application and last-known-good. The Router merges Hub-owned leaves underneath local pins, validates the resulting full Config, persists the Hub overlay and acknowledgment outbox atomically with mode 0600, and publishes one lock-free effective snapshot. A request that already captured the previous Arc<Config> continues unchanged; the next request sees the new policy. Dry run performs the same fetch, bounds, digest, merge, and full validation but does not persist or publish the candidate. Duplicate delivery, a lost result acknowledgment, reconnect, restart, and rollback-as-a-new-revision are idempotent. An unsupported schema/path, stale optimistic revision, immutable/local leaf, malformed or duplicate path, invalid scalar/range, digest mismatch, or oversize response is rejected without changing the active snapshot.

Hub transport failures, timeouts, and response-decode failures are not configuration decisions and therefore never produce a terminal rejection: the agent leaves the assignment unacknowledged and retries while serving last-known-good. Only a deterministic validation decision is reported to POST /api/agent/v1/config/apply-result as applied, validated, rejected, or superseded; results and logs contain bounded path/status metadata, never parameter values. The state file also binds persisted content to the enrolled router identity so one router cannot restore another router's overlay.

Implementation: src/control_plane/config_sync.rs (typed adapter, canonical digest, layered effective store, persistence, snapshot), src/control_plane/client.rs and src/control_plane/agent.rs (outbound transaction and retry), src/control_plane/inventory.rs (masked inventory), and crates/continuum-protocol/src/config.rs (wire contract). Configuration reference: control_plane.config_sync in config.yaml.example.

Token-supply and saturation telemetry¶

The heartbeat and enrollment inventory carry a RouterInventory.supply block plus six BackendInfo fields. They are integer-only and metadata-only. Every reported number comes from real instrumentation: active_requests, accepted_requests, rejected_requests, retried_requests, and fallback_requests use RAII guards and counters at the actual admission, retry, and fallback-service seams (src/control_plane/supply.rs).

accepted_requests and rejected_requests are not a partition. They are two independent pressure counters over overlapping populations, so accepted + rejected is not the number of requests the router saw. A request the admission middleware already counted as accepted can still be refused deeper in the handler (today: hub-policy refusals and batch admission), which counts it in both; the rate limiter also runs app-wide while the accept seam covers only the API routes, so a 429 on /admin counts as rejected without ever counting as accepted. rejected / (accepted + rejected) is therefore not a refusal rate and the ratio can exceed 1.0, deliberately: leaving either seam silent would let a router refusing nearly all of one traffic class report zero pressure, which is worse than an overlap. fallback_requests counts a request once, at the point a fallback backend actually serves the response rather than at selection, through a per-request latch shared by every fallback seam, including a pre-stream chain advance and a mid-stream hop.

An absent optional field means "unknown", never 0. max_concurrency (router and per-backend) is absent because the router enforces no concurrency ceiling. The observed-capacity estimator reports observed_rpm_capacity, observed_input_tpm_capacity, and observed_output_tpm_capacity only after a transient upstream 429 corroborates a full 60-second throughput window with at least two successful completions; token dimensions additionally require complete token accounting. The estimator keeps a 15-minute expiring high-water mark, so cold-start, idle, lightly loaded, token-incomplete, and stale observations stay absent rather than becoming a false zero or demand-shaped capacity. It never reads or merges the operator-declared capacity held by the hub. circuit_state is absent for a backend the breaker has never seen, or whose breaker is disabled: a disabled breaker's get_state() returns Closed, which is a structurally blind value, not a measurement, so it is never reported as Closed. queue_depth and shed_requests are the one exception, reported as a genuine, structural 0 rather than absent, because the router serves straight through with no request queue and no load shedding anywhere.

counters_since_ms is stamped once, at process start (process_start_info()), not refreshed on every heartbeat, so a router that has been up for days reports the same value on every beat; a change in it signals a counter reset (a restart), not steady accumulation. circuit_state is read through a non-materializing peek_state, so a heartbeat never creates the per-backend circuit-breaker entry it is only reading, and never changes what /admin lists. A streaming response moves its in-flight guard into the response body itself, so a backend stays counted until its last SSE frame or the client's disconnect, not merely until the handler returns response headers.

Implementation: src/control_plane/supply.rs (router-level supply tracker), src/control_plane/capacity.rs (per-backend observed-capacity estimator), and src/control_plane/inventory.rs (assembly into the wire types). Wire types: crates/continuum-protocol/src/supply.rs and crates/continuum-protocol/src/heartbeat.rs.

Guardrail telemetry¶

The inventory also carries a RouterInventory.guardrails block: cumulative, metadata-only counters describing what the content-safety guardrails actually did. It mirrors the local Prometheus families (guardrail_checks_total, guardrail_blocks_total, guardrail_verdicts_total, guardrail_stream_buffer_cap_trips_total) so the Hub can see fleet-wide guardrail activity without scraping every router.

The block's field set is closed and typed (continuum_protocol::GuardrailSummary), and it is the Hub's field set: this router adopted the Hub's shape in #1150 after the two were written independently and diverged.

blocks_total and the verdict scalars do not share a denominator. blocks_total counts per provider check, alongside checks_total, so one request two providers both refuse is two blocks. transforms_total and flags_total decompose the mode-applied aggregate, one per stage evaluation, so the same request is one verdict.

The two fail-policy counters report what applied, not what was configured. Monitor mode never gates, so a failed check under monitor is traffic that was served without being inspected and lands in fail_open_total whatever its on_error policy says. fail_closed_total therefore requires both a fail-closed policy and an enforcing mode, which is the only combination that can actually refuse anything. The local Prometheus families keep labelling by the configured policy, because they answer a different question: how a provider is configured, rather than what happened to a request.

errors_total covers both failure surfaces. The service's timeout branch and the hard errors providers report through the Guardrail trait's Result (transport failure, non-success status, unparseable body) record at the same seam, so a provider that fails fast is counted exactly like one that hangs (issue #1175). fail_open_total and fail_closed_total partition errors_total, and a provider answering every request with a connection reset or an HTTP 5xx under the default on_error: fail_open shows up as fail_open_total climbing at the request rate rather than as a heartbeat indistinguishable from a healthy guardrail. A fail-closed hard error stays out of blocks_total / by_category: it is an infrastructure refusal, counted by fail_closed_total, so a moderation outage does not read as a flood of uncategorized unsafe content. The Prometheus surface emits kind="error" for the same events.

Three scalars are derived, not tracked twice. transforms_total, flags_total, and stream_buffer_cap_trips_total are computed from the verdicts and stream_buffer_cap_trips breakdowns in GuardrailSnapshot::into_summary, the single place the derivation happens, so the scalar the Hub reads today and the breakdown a later Hub will read cannot drift apart. A test asserts each total against its sum.

verdicts and stream_buffer_cap_trips ride along additively. The Hub declares neither and does not use deny_unknown_fields, so today's Hub ignores them rather than failing on them, and they become consumable once the Hub adds them.

The counters do not come from the Prometheus registry. The metrics component is optional, so a router built or configured without it has no registry to read, and reporting zeros from such a deployment would tell the Hub that a guardrail which is actively blocking traffic is idle. A separate in-process tracker (src/control_plane/guardrail.rs) is written at the same seams that emit the metrics, each behind its own feature gate, so inventory stays honest with Prometheus off.

Absent means "guardrails are off", zeros mean "on and quiet". The block is reported whenever a guardrail service exists, even before the first check runs, and omitted entirely when the router runs no guardrails. Collapsing the two into one absent case would make a misconfigured router indistinguishable from a quiet one, which is the reason to report this at all.

The label space is closed, and that is a correctness property. Guardrail categories reach the router from third-party moderation providers through GuardrailCategory::Other(String), so a map keyed by that string would let one misbehaving provider expand the Hub's stored key space without bound and carry provider-authored text with it. Each breakdown is instead a fixed-size array of atomics indexed by a closed label tuple: blocks by bounded category id, verdicts by (stage, mode, result), and streaming buffer-cap trips by (strategy, outcome). Any category outside the known vocabulary folds into a single other bucket before it is stored, and an unrecognized stage, mode, or result folds onto an unknown slot rather than being dropped, so closing the space costs no blind spot. The reserved match-list pseudo-providers (match_list_deny / match_list_allow) need no special case: their deny verdicts carry the ordinary deny_list category.

by_category is a JSON object because that is the Hub's shape, but its keys are still not free text: the only way a key enters it is GuardrailCategoryId::as_str(), a closed set of twelve compile-time literals, each inside the Hub's ingestion bounds on count (32), byte length (64), and charset ([a-z][a-z0-9_]*). GuardrailSummary::is_bounded() checks all three, and the router's own emission ceiling is asserted against the Hub's ingestion ceiling at compile time.

Privacy. Counts and bounded ids only: no prompt text, no completion text, no matched span, no matched rule, and no redacted-PII value. This holds on the error path too: errors_total and the two fail-policy counters carry no provider name and no failure text. An integration test asserts this against a serialized inventory rather than against the types.

counters_since_ms follows the SupplySummary contract and is stamped from the same process_start_info() value, so both counter families reset together and the Hub reads one restart rather than two.

The Hub's own wire fixture is pinned here. crates/continuum-protocol/tests/fixtures/guardrail_summary.json is a byte-for-byte copy of the Hub's committed fixture, pinned by crates/continuum-protocol/tests/guardrail_summary_hub_fixture.rs in both directions and re-checked against a live tracker read in tests/control_plane_guardrail_telemetry_test.rs. Both repositories previously round-tripped only their own types, which is exactly why the shapes could diverge while both stayed green. Never regenerate that fixture from local types: when the Hub moves it, re-copy it and update the provenance in the same change.

Implementation: src/control_plane/guardrail.rs (counter tracker), src/control_plane/inventory.rs (assembly and the report/omit decision), and the recording seams in src/services/guardrail/service.rs and src/services/guardrail/gate.rs. Wire types: crates/continuum-protocol/src/guardrail_telemetry.rs and crates/continuum-protocol/src/heartbeat.rs.

Policy sync and local enforcement¶

When control_plane.policy.enabled is set (and the control-plane feature is compiled and the agent is enabled), the agent keeps a hub policy snapshot current and the router enforces it locally, after API-key authentication, on every API request.

Delivery paths. Two paths feed the same in-memory policy store, both carrying full PolicyEnvelopes (the org's API key table as hashes, tier limits, per-key month-to-date budget snapshots, and agent settings). Full envelopes make application idempotent: any cursor gap is healed by the next envelope.

• Poll (POST /api/agent/v1/policy/sync): runs on policy.poll_interval_secs (default 5s) for as long as enforcement is enabled. This is the correctness core: on its own it bounds revocation and budget propagation by the poll interval.
• Stream (GET /api/agent/v1/policy/stream?cursor=<opaque>): a WebSocket held open for push-style delivery, so a revocation propagates in under five seconds while connected. Every frame is one complete envelope as JSON text. On disconnect the client reconnects with backoff, presenting the last applied cursor; the poll loop covers the gap in the meantime. Disable with policy.stream_enabled: false.

Bounded envelope decode. Both delivery paths refuse an envelope over a derived byte ceiling (~152.6 MiB, MAX_POLICY_SYNC_RESPONSE_BYTES in src/control_plane/client.rs) instead of buffering it without bound: the poll path pre-checks the declared Content-Length and then a running chunk total, and the WebSocket stream sets its message and frame ceilings to the same constant, so the two paths agree by decision rather than by tungstenite's smaller 64 MiB library default (the frame ceiling moves with the message ceiling because the Hub delivers each envelope as one unfragmented text frame). The ceiling is a documented term-by-term sum over the Hub's own distribution budgets (a 100 MiB key-table allowance for the one member with no Hub-side per-org ceiling, sized in the code comment against cumulative rows including revoked and expired keys, which the Hub keeps forever by default; 1,024 tiers with the 4 MiB aggregate request-params budget and character-capped unicode allowlist names; the 256 KiB guardrail canonical budget; 5,000 cost-center assignments plus 1,024 capped centers; and explicit allowances for the members the Hub does not bound at all), sized so a legitimate tenant at every ceiling at once still decodes. A refusal is loud, never silent: it is a distinct typed error (HubError::ResponseTooLarge, never confused with a parse failure), logs at error! with the limit and the observed size (never body content), and increments control_plane_policy_sync_failure_total{reason="envelope_too_large"}, separate from reason="other" for ordinary sync failures, because a persistent over-limit refusal leaves a restarted router permanently fail-open and must be distinguishable from a hub that is temporarily down. Both reason series are instantiated at zero at registration, so a fleet alert sees the series on routers that have refused nothing yet, and the refusal count with its last-occurrence time also appears as policy_sync.oversize_refusals / last_oversize_refusal_ms on GET /admin/control-plane/status, where a nonzero count beside a null last_sync_ms is the cold-start fail-open signature. The store is untouched by a refusal: the last-known-good snapshot keeps enforcing, and no key or limit ever widens. The smaller Hub response decodes (enroll, usage acks, probe acks) carry their own derived bounds with the same typed refusal; an unreadable oversize usage ack is terminal for its already-processed batch rather than retried forever, so the push loop keeps making progress.

What is enforced. For a presented API key whose SHA-256 matches a synced KeyEntry (constant-time compare; plaintext keys never travel over the agent protocol):

• Revocation: a revoked entry is rejected immediately (revocation, expiry, and org suspension all arrive as this flag).
• Rate limits: per-tier rpm as a fixed 60-second window (the request that would exceed it gets 429); input_tpm and output_tpm as post-response windows fed by real token counts at the usage taps, so the request following the one that filled the window is rejected. Cache-read input tokens do not count toward input_tpm.
• Monthly budgets: tokens_per_month / requests_per_month are enforced against the hub's per-key KeyUsageSnapshot (month-to-date as ingested by the hub) plus a locally observed delta since that snapshot. The local delta resets when a fresher snapshot arrives or the UTC month rolls over. Overspend is therefore bounded by the sync interval by design (hub ARCHITECTURE.md section 4).
• Model allowlist: a request for a model outside the tier's model_allowlist is rejected with 403; an absent allowlist means all served models.
• Request-parameter limits: a tier may bound temperature, top_p, max_tokens, presence_penalty, frequency_penalty, top_k, and min_p, with optional exact alias-resolved model scopes. Decimal bounds cross the wire as integer microunits. Every ingress captures one client-requested identity after local alias resolution and applies the same models/ prefix normalization and exact backend alias mapping used by routing. The Router intersects Hub limits with deployment-local request_params limits, rejects an empty intersection, and freezes the selected policy before cache lookup, substitution, arbitrage, backend selection, retry, or fallback. The same effective request is used by Chat Completions, Completions, Responses, and Anthropic Messages.
• Headers: enforced responses carry x-continuum-ratelimit-limit/-remaining/-reset for the most constrained dimension plus a suffixed triple per configured dimension (-rpm, -input-tpm, -output-tpm, -monthly-tokens, -monthly-requests); 429s also carry retry-after.

Atomic tier-policy state. A full snapshot is validated and converted before the live ArcSwap changes. Invalid ranges, duplicate tier ids, stale snapshots, and tier-digest mismatches preserve the complete last-known-good snapshot. Poll requests capture the active acceptance generation before network I/O, so a delayed same-millisecond response cannot overwrite a stream revision accepted while that request was in flight; later deliveries from that accepted generation remain valid even when the Hub clock timestamp is unchanged. Heartbeats read capability/status evidence and the applied cursor from one atomic publication, preventing cross-generation acknowledgment pairs. RouterInventory.policy_status advertises request_param_limits_v1, acknowledges a successful Hub-provided cursor/digest pair, and can report one stable rejection code without a free-form value. A successful later snapshot, including an authoritative request_params clear, replaces the old limits and clears the rejection. If a snapshot omits the optional digest, its limits are enforced but the Router does not invent active digest evidence.

Guardrail-policy acknowledgment. RouterInventory.policy_status carries one capability list built in a single place (PolicyStore::capabilities), so no status-writing path can advertise a set the others do not. guardrail_policy_v1 joins request_param_limits_v1 in that list only once a live guardrail-policy reconciler has attached to the policy store, because the capability is a promise about execution rather than a build flag: a router that would receive, validate, and store a hub guardrail policy without ever running it advertises nothing, and the Hub derives unsupported instead of waiting for an acknowledgment that will never arrive.

Receipt and enforcement are acknowledged separately. Accepting an envelope records the guardrail digest the wire layer applied, which is receipt only; whether that policy could be composed into a running guardrail service is decided afterwards by the reconciler. When the reconciler refuses the exact digest the active revision would claim, the claim is withdrawn and the same digest is reported as rejected with guardrails_unavailable instead, so a Fleet view never shows governance in effect while nothing enforces it. The guardrail and tier tracks stay independent in both directions: a refused guardrail body neither discards the tier policy from the same envelope nor blocks its acknowledgment, and a tier rejection says nothing about a guardrail body. An authoritative clear acknowledges the canonical cleared-policy digest rather than reporting nothing, so a cleared tenant converges instead of sitting at pending or stale. Because the store keeps an unexecutable policy applied at the wire level, a later local edit that adds the missing wiring lets the same body compose cleanly and restores the active claim without a new envelope.

Stable backend identity. stable_backend_identity_v1 also sits in that list, and unconditionally, because unlike guardrail policy it has no separate executor that may or may not attach: config validation, heartbeat inventory, and the usage taps are compiled and wired as one unit, so there is no reachable state in which an accepted backends[].backend_id is dropped by one of the three. The capability describes the implementation, not the data. A router whose operator has configured no ids still advertises it and simply reports none, which is exactly the "can report, nothing to report" state the Hub must be able to tell apart from "cannot report at all".

The identity itself is operator-authored and never derived. BackendInfo.name is a mutable display label and a (provider, model) pair is not a backend, so deriving an identity from either would merge two distinct cost owners that share a pair, or split one cost owner across a rename. Each usage tap resolves the id at its serving seam, against the config view the selection actually used, and carries it on the UsageEvent rather than letting the push loop look it up by name later, because a hot reload in that window would rename the backend and silently drop the attribution of usage already served. A fallback hop overwrites it just as it overwrites the TTFT origin, a provider batch carries the id captured at submit, and a local cache hit dispatched no backend and so reports nothing. Inventory resolves through the same function, so the ids an operator discovers there are the ids their usage is charged to.

The capability strings a router can advertise, request_param_limits_v1, guardrail_policy_v1, and stable_backend_identity_v1, are what let the Hub derive a propagation state from bounded evidence rather than from a free-form report: