Library Usage¶

Continuum Router can be used as an embeddable Rust library crate in addition to the standalone CLI binary. This enables you to integrate LLM routing directly into your Rust application without spawning a separate process.

Use Cases¶

• Embedding in existing applications: Integrate LLM routing into your Axum/Tokio application
• Programmatic configuration: Configure the router entirely in Rust code without YAML files
• Custom extensions: Add custom middleware, routes, or modify behavior around the router
• Integration testing: Create router instances programmatically for test suites

Adding the Dependency¶

Add continuum-router as a library dependency in your Cargo.toml:

[dependencies]
continuum-router = { git = "https://github.com/lablup/continuum-router.git" }
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
axum = "0.7"

To reduce compile times and binary size, you can disable default features and select only what you need:

[dependencies]
continuum-router = { git = "https://github.com/lablup/continuum-router.git", default-features = false, features = ["metrics", "hot-reload"] }

See Cargo Feature Flags for the full list of available features.

Quick Start¶

Standalone Server from Config File¶

The simplest way to use the library is to load a YAML config file and run the server:

use continuum_router::ContinuumRouter;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize tracing (optional - the library does not initialize tracing)
    tracing_subscriber::fmt::init();

    let router = ContinuumRouter::from_config_file("config.yaml")
        .await?
        .enable_hot_reload(true)
        .build()
        .await?;

    router.serve("0.0.0.0:8080").await?;
    Ok(())
}

Standalone Server from Programmatic Config¶

You can also configure the router entirely in code. BackendConfig implements no Default, so a struct literal has to name every field; there is no ..Default::default() shorthand. If that is too verbose, skip to the type-safe builder below, which is the recommended way to build backends by hand. A compiled version of this snippet lives at examples/programmatic_config.rs.

use continuum_router::{ContinuumRouter, Config};
use continuum_router::config::{BackendConfig, BackendTypeConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    tracing_subscriber::fmt::init();

    let mut config = Config::default();
    config.backends = vec![
        BackendConfig {
            name: "openai".to_string(),
            backend_id: None,
            backend_type: BackendTypeConfig::Openai,
            url: "https://api.openai.com/v1".to_string(),
            weight: 1,
            api_key: std::env::var("OPENAI_API_KEY").ok(),
            org_id: None,
            auth: None,
            models: vec!["gpt-4o".to_string()],
            model_configs: vec![],
            retry_override: None,
            health_check: None,
            anthropic_auto_cache_control: None,
            anthropic_fast_mode: None,
            endpoint_type: None,
            region: None,
            role: Default::default(),
            external_storage: None,
            internal: false,
        },
        BackendConfig {
            name: "ollama".to_string(),
            backend_id: None,
            backend_type: BackendTypeConfig::Ollama,
            url: "http://localhost:11434".to_string(),
            weight: 1,
            api_key: None,
            org_id: None,
            auth: None,
            models: vec![],  // Auto-discovered
            model_configs: vec![],
            retry_override: None,
            health_check: None,
            anthropic_auto_cache_control: None,
            anthropic_fast_mode: None,
            endpoint_type: None,
            region: None,
            role: Default::default(),
            external_storage: None,
            internal: false,
        },
    ];

    let router = ContinuumRouter::from_config(config)
        .enable_health_checks(true)
        .build()
        .await?;

    router.serve("0.0.0.0:8080").await?;
    Ok(())
}

Type-Safe Config Builders¶

For a cleaner alternative to constructing BackendConfig structs manually, use the type-safe builder API. Builders validate all settings at build time and provide provider-specific constructors:

use continuum_router::config::builder::{BackendConfigBuilder, ConfigBuilder};
use continuum_router::ContinuumRouter;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    tracing_subscriber::fmt::init();

    let openai = BackendConfigBuilder::openai("https://api.openai.com/v1", "sk-...")
        .name("primary-openai")
        .models(vec!["gpt-4o", "gpt-4o-mini"])
        .build()?;

    let ollama = BackendConfigBuilder::ollama("http://localhost:11434")
        .build()?;

    let config = ConfigBuilder::new()
        .add_backend(openai)
        .add_backend(ollama)
        .bind_address("0.0.0.0:8080")
        .enable_health_checks(true)
        .logging_level("info")
        .build()?;

    let router = ContinuumRouter::from_config(config)
        .build()
        .await?;

    router.serve("0.0.0.0:8080").await?;
    Ok(())
}

See the full Rust Builder API reference for all available builder methods, provider constructors, and error types.

Embed in an Existing Axum Application¶

Use into_router() to get an Axum Router that you can nest into your own application:

use axum::{routing::get, Router};
use continuum_router::ContinuumRouter;

async fn custom_handler() -> &'static str {
    "Custom endpoint"
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    tracing_subscriber::fmt::init();

    // Build the Continuum Router
    let llm_router = ContinuumRouter::from_config_file("config.yaml")
        .await?
        .build()
        .await?;

    // Nest it under /llm alongside your own routes
    let app = Router::new()
        .route("/custom", get(custom_handler))
        .nest("/llm", llm_router.into_router());

    let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await?;
    axum::serve(listener, app).await?;
    Ok(())
}

When embedded, the router's endpoints become available under the nest prefix. For example, /llm/v1/chat/completions, /llm/v1/models, /llm/admin/backends, etc.

In-Process Embedding for iOS and Mobile¶

When the host process cannot run the router as a sidecar subprocess (for example on iOS, where the app sandbox forbids spawning a child process), link continuum-router as a Rust library and run its HTTP server in-process on a loopback address that an in-app WebView targets.

Use the minimal embed feature so the dependency graph stays free of crates that are awkward or impossible to cross-compile for mobile targets:

[dependencies]
continuum-router = { git = "https://github.com/lablup/continuum-router.git", default-features = false, features = ["embed"] }
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }

The embed feature enables no extra component and pulls in none of the iOS-hostile optional dependencies (rusqlite, redis, prometheus, rust-embed, aws-*, clap, notify, rmcp, object_store, gcp_auth). gcp_auth is excluded because it is only needed for Gemini-via-Vertex; the public Gemini API (generativelanguage.googleapis.com, API key) path does not require it.

The entrypoint is serve_embedded, a thin async wrapper over the canonical builder path that is callable without the cli feature:

use continuum_router::{serve_embedded, Config};

async fn run(config: Config) -> Result<(), Box<dyn std::error::Error>> {
    // Blocks until SIGINT/SIGTERM; on mobile the host controls the runtime.
    serve_embedded(config, "127.0.0.1:8080").await?;
    Ok(())
}

serve_embedded(config, addr) is exactly equivalent to:

use continuum_router::{Config, ContinuumRouter};

async fn run(config: Config, addr: &str) -> Result<(), Box<dyn std::error::Error>> {
    ContinuumRouter::from_config(config)
        .build()
        .await?
        .serve(addr)
        .await?;
    Ok(())
}

Use the canonical from_config(...).build() path with into_router() instead when you need finer lifecycle control (your own listener, graceful shutdown, or nesting under an existing Axum app). Verify the trimmed dependency graph with:

cargo tree --no-default-features --features embed --edges normal --target aarch64-apple-ios

Builder API Reference¶

The builder is created via ContinuumRouter::from_config() or ContinuumRouter::from_config_file(), and configured with a fluent method chain before calling .build().await.

Entry Points¶

ContinuumRouter::from_config(config: Config) -> ContinuumRouterBuilder¶

Create a builder from a programmatic Config struct. No file watching or hot-reload is available unless a ConfigManager is attached manually via with_config_manager().

ContinuumRouter::from_config_file(path: &str) -> Result<ContinuumRouterBuilder, RouterError>¶

Create a builder by loading configuration from a YAML/TOML file. This internally creates a ConfigManager that supports hot-reload when enabled. The config directory is automatically derived from the file path for prompt file resolution.

Builder Methods¶

All builder methods return Self for chaining.

ContinuumRouter Methods¶

After building, you get a ContinuumRouter instance with these methods:

Hot-Reload in Library Mode¶

Continuum Router's hot-reload infrastructure works in library mode just as it does for the CLI binary.

File-Based Hot-Reload¶

When you use from_config_file() with hot-reload enabled, the router watches the config file for changes and automatically applies updates:

let router = ContinuumRouter::from_config_file("config.yaml")
    .await?
    .enable_hot_reload(true)
    .build()
    .await?;

Valid changes to config.yaml are detected through filesystem notifications and published as a new configuration snapshot. Backends and health checks update gradually; active rate-limit, prompt, request-parameter, streaming, smart-routing, and selected policy services consume supported dynamic fields. Startup-built services keep their startup values until restart.

Programmatic runtime publication¶

config_handle() returns a shared read/subscription handle. It does not expose the mutable ConfigManager::set_config operation. Use the Admin section APIs, an import with apply: true, or update the watched file to publish a runtime revision.

Subscribing to Config Changes¶

You can subscribe to configuration change notifications using the watch channel pattern:

if let Some(config_handle) = router.config_handle() {
    let mut rx = config_handle.subscribe();

    tokio::spawn(async move {
        while rx.changed().await.is_ok() {
            let config = rx.borrow().clone();
            println!("Config updated: {} backends", config.backends.len());
        }
    });
}

Hot-reloadable settings¶

Use the conservative runtime classification:

Immediate effect when the corresponding service exists:

• Rate limiting and proxy data-plane circuit policy
• Global prompts and request-parameter policy
• Streaming and smart-routing policy snapshots
• Selection strategy, retry policy, and per-request timeout budgets
• api_keys.mode, all prefix-routing fields, fallback chains, and fallback policy

Gradual effect:

• Backend add/remove/modify with graceful draining
• Health-check settings

Requires restart or has no live consumer:

• Server listeners, workers, connection-pool construction, and CORS
• Logging and tracing construction
• timeouts.connection and timeouts.request.streaming.total (shared HTTP-client ceilings; per-request budgets still reload live)
• File-watcher API-key material and startup-built cache/service objects

Check /admin/config/hot-reload-status for the running build.

Accessing Internal State¶

The state() method provides access to the shared AppState, which holds references to all internal components:

let router = ContinuumRouter::from_config_file("config.yaml")
    .await?
    .build()
    .await?;

let state = router.state();

// Check health checker status
if let Some(ref checker) = state.health_checker {
    let status = checker.get_backend_health_status().await;
    for backend in &status {
        println!("{}: {:?}", backend.name, backend.status);
    }
}

// Access the current config
let config = state.current_config();
println!("Selection strategy: {:?}", config.selection_strategy);

Custom HTTP Client¶

You can provide a pre-configured reqwest::Client for backend communication. This is useful when you need specific TLS settings, proxy configuration, or connection pool tuning:

let client = reqwest::Client::builder()
    .timeout(std::time::Duration::from_secs(60))
    .connect_timeout(std::time::Duration::from_secs(5))
    .pool_max_idle_per_host(32)
    .build()?;

let router = ContinuumRouter::from_config(config)
    .with_http_client(client)
    .build()
    .await?;

Important Notes¶

Tracing Initialization¶

The library does not initialize tracing. You must call tracing_subscriber::fmt::init() (or your preferred tracing setup) before building the router if you want log output.

Metrics Singleton¶

The Prometheus metrics registry (RouterMetrics) is a process-global singleton. If you embed multiple ContinuumRouter instances in the same process, they share the same metrics registry.

Graceful Shutdown¶

When using serve(), the router handles SIGINT and SIGTERM signals automatically and performs graceful shutdown (draining connections, cleaning up Unix sockets, stopping background tasks).

When using into_router() for embedding, you manage the server lifecycle yourself. Call shutdown() on the ContinuumRouter instance when you're done, or simply drop it to release resources.

Runnable Examples¶

The repository includes four ready-to-run examples in the examples/ directory. Each example can be executed with cargo run --example <name>.

standalone_server¶

Minimal standalone server that loads a config file and starts serving:

cargo run --example standalone_server -- --config config.yaml

See examples/standalone_server.rs for the full source.

axum_integration¶

Embeds the router under /llm in a larger Axum application:

cargo run --example axum_integration -- --config config.yaml

After starting, custom endpoints are available at /health and /info, while LLM endpoints are under /llm/v1/....

See examples/axum_integration.rs for the full source.

programmatic_config¶

Configures the router entirely in Rust code without a YAML file:

OPENAI_API_KEY=sk-... cargo run --example programmatic_config
# or with a local Ollama instance:
OLLAMA_URL=http://localhost:11434 cargo run --example programmatic_config

See examples/programmatic_config.rs for the full source.

hot_reload¶

Demonstrates file-based hot-reload and subscribing to config-change events:

cargo run --example hot_reload -- --config config.yaml

While the server is running, edit config.yaml to see changes applied automatically.

See examples/hot_reload.rs for the full source.

ACP Support¶

The ACP (Agent Communication Protocol) module is always compiled into the library crate (pub mod acp). When building custom applications, you can access ACP types, handlers, and transport implementations directly:

use continuum_router::acp::config::AcpConfig;
use continuum_router::acp::router::AcpMethodRouter;
use continuum_router::acp::session::SessionStore;
use continuum_router::acp::transport::stdio::StdioTransport;

The CLI binary's --mode stdio implementation in src/main.rs serves as a reference for how to wire ACP handlers into a custom application. See the ACP Architecture documentation for protocol details and module structure.

Further Reading¶

• Architecture Guide for the internal design of the server module
• Configuration Guide for the full YAML configuration reference
• ACP Usage Guide for IDE and tool integration via stdio transport
• API Reference for the OpenAI-compatible HTTP API
• Development Guide for contributing to the project and the examples/ directory