Library Usage¶

Continuum Router can be used as an embeddable Rust library crate in addition to the standalone CLI binary. This enables you to integrate LLM routing directly into your Rust application without spawning a separate process.

Use Cases¶

• Embedding in existing applications: Integrate LLM routing into your Axum/Tokio application
• Programmatic configuration: Configure the router entirely in Rust code without YAML files
• Custom extensions: Add custom middleware, routes, or modify behavior around the router
• Integration testing: Create router instances programmatically for test suites

Adding the Dependency¶

Add continuum-router as a library dependency in your Cargo.toml:

[dependencies]
continuum-router = { git = "https://github.com/lablup/continuum-router.git" }
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
axum = "0.7"

To reduce compile times and binary size, you can disable default features and select only what you need:

[dependencies]
continuum-router = { git = "https://github.com/lablup/continuum-router.git", default-features = false, features = ["metrics", "hot-reload"] }

See Cargo Feature Flags for the full list of available features.

Quick Start¶

Standalone Server from Config File¶

The simplest way to use the library is to load a YAML config file and run the server:

use continuum_router::ContinuumRouter;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize tracing (optional - the library does not initialize tracing)
    tracing_subscriber::fmt::init();

    let router = ContinuumRouter::from_config_file("config.yaml")
        .await?
        .enable_hot_reload(true)
        .build()
        .await?;

    router.serve("0.0.0.0:8080").await?;
    Ok(())
}

Standalone Server from Programmatic Config¶

You can also configure the router entirely in code:

use continuum_router::{ContinuumRouter, Config};
use continuum_router::config::{BackendConfig, BackendTypeConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    tracing_subscriber::fmt::init();

    let mut config = Config::default();
    config.backends = vec![
        BackendConfig {
            name: "openai".to_string(),
            url: "https://api.openai.com/v1".to_string(),
            backend_type: BackendTypeConfig::Openai,
            api_key: std::env::var("OPENAI_API_KEY").ok(),
            models: vec!["gpt-4o".to_string()],
            ..BackendConfig::default()
        },
        BackendConfig {
            name: "ollama".to_string(),
            url: "http://localhost:11434".to_string(),
            backend_type: BackendTypeConfig::Ollama,
            models: vec![],  // Auto-discovered
            ..BackendConfig::default()
        },
    ];

    let router = ContinuumRouter::from_config(config)
        .enable_health_checks(true)
        .enable_circuit_breaker(true)
        .build()
        .await?;

    router.serve("0.0.0.0:8080").await?;
    Ok(())
}

Type-Safe Config Builders¶

For a cleaner alternative to constructing BackendConfig structs manually, use the type-safe builder API. Builders validate all settings at build time and provide provider-specific constructors:

use continuum_router::config::builder::{BackendConfigBuilder, ConfigBuilder};
use continuum_router::ContinuumRouter;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    tracing_subscriber::fmt::init();

    let openai = BackendConfigBuilder::openai("https://api.openai.com/v1", "sk-...")
        .name("primary-openai")
        .models(vec!["gpt-4o", "gpt-4o-mini"])
        .build()?;

    let ollama = BackendConfigBuilder::ollama("http://localhost:11434")
        .build()?;

    let config = ConfigBuilder::new()
        .add_backend(openai)
        .add_backend(ollama)
        .bind_address("0.0.0.0:8080")
        .enable_health_checks(true)
        .logging_level("info")
        .build()?;

    let router = ContinuumRouter::from_config(config)
        .build()
        .await?;

    router.serve("0.0.0.0:8080").await?;
    Ok(())
}

See the full Rust Builder API reference for all available builder methods, provider constructors, and error types.

Embed in an Existing Axum Application¶

Use into_router() to get an Axum Router that you can nest into your own application:

use axum::{routing::get, Router};
use continuum_router::ContinuumRouter;

async fn custom_handler() -> &'static str {
    "Custom endpoint"
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    tracing_subscriber::fmt::init();

    // Build the Continuum Router
    let llm_router = ContinuumRouter::from_config_file("config.yaml")
        .await?
        .build()
        .await?;

    // Nest it under /llm alongside your own routes
    let app = Router::new()
        .route("/custom", get(custom_handler))
        .nest("/llm", llm_router.into_router());

    let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await?;
    axum::serve(listener, app).await?;
    Ok(())
}

When embedded, the router's endpoints become available under the nest prefix. For example, /llm/v1/chat/completions, /llm/v1/models, /llm/admin/backends, etc.

Builder API Reference¶

The builder is created via ContinuumRouter::from_config() or ContinuumRouter::from_config_file(), and configured with a fluent method chain before calling .build().await.

Entry Points¶

ContinuumRouter::from_config(config: Config) -> ContinuumRouterBuilder¶

Create a builder from a programmatic Config struct. No file watching or hot-reload is available unless a ConfigManager is attached manually via with_config_manager().

ContinuumRouter::from_config_file(path: &str) -> Result<ContinuumRouterBuilder, RouterError>¶

Create a builder by loading configuration from a YAML/TOML file. This internally creates a ConfigManager that supports hot-reload when enabled. The config directory is automatically derived from the file path for prompt file resolution.

Builder Methods¶

All builder methods return Self for chaining.

ContinuumRouter Methods¶

After building, you get a ContinuumRouter instance with these methods:

Hot-Reload in Library Mode¶

Continuum Router's hot-reload infrastructure works in library mode just as it does for the CLI binary.

File-Based Hot-Reload¶

When you use from_config_file() with hot-reload enabled, the router watches the config file for changes and automatically applies updates:

let router = ContinuumRouter::from_config_file("config.yaml")
    .await?
    .enable_hot_reload(true)
    .build()
    .await?;

Any changes to config.yaml are detected via filesystem notifications and propagated to all components (backends, health checker, circuit breaker, rate limiter, model service).

Programmatic Runtime Updates¶

You can also update configuration at runtime via the ConfigManager:

let router = ContinuumRouter::from_config_file("config.yaml")
    .await?
    .enable_hot_reload(true)
    .build()
    .await?;

if let Some(config_handle) = router.config_handle() {
    // Get current config
    let mut new_config = config_handle.get_config().await;

    // Modify it
    new_config.backends.push(BackendConfig {
        name: "new-backend".to_string(),
        url: "http://localhost:8000".to_string(),
        ..BackendConfig::default()
    });

    // Apply the update (triggers HotReloadService)
    config_handle.set_config(new_config).await?;
}

Subscribing to Config Changes¶

You can subscribe to configuration change notifications using the watch channel pattern:

if let Some(config_handle) = router.config_handle() {
    let mut rx = config_handle.subscribe();

    tokio::spawn(async move {
        while rx.changed().await.is_ok() {
            let config = rx.borrow().clone();
            println!("Config updated: {} backends", config.backends.len());
        }
    });
}

Hot-Reloadable Settings¶

Not all settings can be changed at runtime. Here is what supports hot-reload:

Immediate effect:

• Logging level
• Rate limiting settings
• Circuit breaker settings
• Retry settings
• Global prompts

Gradual effect (new connections use new config):

• Backend add/remove/modify (with graceful draining)
• Health check settings
• Timeout settings
• Selection strategy

Requires restart:

• Server bind address
• Worker count
• Connection pool size

Accessing Internal State¶

The state() method provides access to the shared AppState, which holds references to all internal components:

let router = ContinuumRouter::from_config_file("config.yaml")
    .await?
    .build()
    .await?;

let state = router.state();

// Check health checker status
if let Some(ref checker) = state.health_checker {
    let status = checker.get_backend_health_status().await;
    for backend in &status {
        println!("{}: {:?}", backend.name, backend.status);
    }
}

// Access the current config
let config = state.current_config();
println!("Selection strategy: {:?}", config.selection_strategy);

Custom HTTP Client¶

You can provide a pre-configured reqwest::Client for backend communication. This is useful when you need specific TLS settings, proxy configuration, or connection pool tuning:

let client = reqwest::Client::builder()
    .timeout(std::time::Duration::from_secs(60))
    .connect_timeout(std::time::Duration::from_secs(5))
    .pool_max_idle_per_host(32)
    .build()?;

let router = ContinuumRouter::from_config(config)
    .with_http_client(client)
    .build()
    .await?;

Important Notes¶

Tracing Initialization¶

The library does not initialize tracing. You must call tracing_subscriber::fmt::init() (or your preferred tracing setup) before building the router if you want log output.

Metrics Singleton¶

The Prometheus metrics registry (RouterMetrics) is a process-global singleton. If you embed multiple ContinuumRouter instances in the same process, they will share the same metrics registry. This is a known limitation that may be addressed in a future release.

Graceful Shutdown¶

When using serve(), the router handles SIGINT and SIGTERM signals automatically and performs graceful shutdown (draining connections, cleaning up Unix sockets, stopping background tasks).

When using into_router() for embedding, you manage the server lifecycle yourself. Call shutdown() on the ContinuumRouter instance when you're done, or simply drop it to release resources.

Runnable Examples¶

The repository includes four ready-to-run examples in the examples/ directory. Each example can be executed with cargo run --example <name>.

standalone_server¶

Minimal standalone server that loads a config file and starts serving:

cargo run --example standalone_server -- --config config.yaml

See examples/standalone_server.rs for the full source.

axum_integration¶

Embeds the router under /llm in a larger Axum application:

cargo run --example axum_integration -- --config config.yaml

After starting, custom endpoints are available at /health and /info, while LLM endpoints are under /llm/v1/....

See examples/axum_integration.rs for the full source.

programmatic_config¶

Configures the router entirely in Rust code without a YAML file:

OPENAI_API_KEY=sk-... cargo run --example programmatic_config
# or with a local Ollama instance:
OLLAMA_URL=http://localhost:11434 cargo run --example programmatic_config

See examples/programmatic_config.rs for the full source.

hot_reload¶

Demonstrates file-based hot-reload and subscribing to config-change events:

cargo run --example hot_reload -- --config config.yaml

While the server is running, edit config.yaml to see changes applied automatically.

See examples/hot_reload.rs for the full source.

ACP Support¶

The ACP (Agent Communication Protocol) module is always compiled into the library crate (pub mod acp). When building custom applications, you can access ACP types, handlers, and transport implementations directly:

use continuum_router::acp::config::AcpConfig;
use continuum_router::acp::router::AcpMethodRouter;
use continuum_router::acp::session::SessionStore;
use continuum_router::acp::transport::stdio::StdioTransport;

The CLI binary's --mode stdio implementation in src/main.rs serves as a reference for how to wire ACP handlers into a custom application. See the ACP Architecture documentation for protocol details and module structure.

Further Reading¶

• Architecture Guide for the internal design of the server module
• Configuration Guide for the full YAML configuration reference
• ACP Usage Guide for IDE and tool integration via stdio transport
• API Reference for the OpenAI-compatible HTTP API
• Development Guide for contributing to the project and the examples/ directory