Skip to content

Development Guide

This guide covers the development setup, architecture, testing, and contribution guidelines for the Continuum Router project.

Table of Contents

Prerequisites

Required Tools

  • Rust: 1.95 or later (Install Rust); the root router package and standalone perf/ harness use Rust edition 2024, while this repository pins Rust 1.97.1 in rust-toolchain.toml for reproducible CI and local linting.
  • Git: For version control
  • C Compiler: gcc, clang, or MSVC (for native dependencies)

Rust edition, MSRV, and formatting

The root continuum-router package and standalone perf/ harness declare Rust edition 2024 with rust-version = "1.95". The vendored crates/continuum-protocol package intentionally stays on edition 2021 and Rust 1.88 because it mirrors the upstream Continuum Hub protocol crate; Cargo supports this mixed-edition graph, and router CI validates that path dependency separately. rustfmt.toml pins style_edition = "2021" so cargo fmt and editor integrations keep the existing formatting style until a separate formatting-only migration is accepted.

  • cargo-watch: For auto-recompilation during development
  • cargo-edit: For managing dependencies
  • cargo-audit: For security vulnerability checking
  • cargo-flamegraph: For performance profiling
  • rust-analyzer: IDE support (VS Code, IntelliJ, vim/neovim)

Install Development Tools

# Install essential development tools
cargo install cargo-watch cargo-edit cargo-audit cargo-flamegraph

# Install code coverage tools
cargo install cargo-tarpaulin

# Install benchmarking tools
cargo install cargo-criterion

Development Setup

Clone and Setup

# Clone the repository
git clone https://github.com/lablup/backend.ai-continuum.git
cd backend.ai-continuum/continuum-router

# Check that everything compiles
cargo check

# Run initial test suite
cargo test

# Build development version
cargo build

IDE Setup

VS Code

  1. Install the rust-analyzer extension
  2. Install the CodeLLDB extension for debugging
  3. Use the provided .vscode/settings.json:
{
  "rust-analyzer.cargo.features": "all",
  "rust-analyzer.checkOnSave.command": "clippy",
  "editor.formatOnSave": true,
  "[rust]": {
    "editor.defaultFormatter": "rust-lang.rust-analyzer"
  }
}

IntelliJ IDEA / CLion

  1. Install the Rust plugin
  2. Open the project root
  3. Configure Cargo settings in Preferences → Build → Cargo

Vim/Neovim

  1. Install rust-analyzer via Mason or manually
  2. Configure LSP client (nvim-lspconfig, coc.nvim, etc.)
  3. Example nvim-lspconfig setup:
require('lspconfig').rust_analyzer.setup({
  settings = {
    ["rust-analyzer"] = {
      cargo = { features = "all" },
      checkOnSave = { command = "clippy" }
    }
  }
})

Project Structure

Layer Architecture

The project follows a clean 4-layer architecture:

Directory Structure

src/
├── main.rs                 # Application entry point and CLI
├── lib.rs                  # Library exports and module declarations
│
├── core/                   # Core domain layer (no external dependencies)
│   ├── mod.rs             # Core module exports
│   ├── models/            # Domain models
│   │   ├── backend.rs     # Backend configuration and state
│   │   ├── model.rs       # LLM model representations
│   │   ├── request.rs     # Request/response models
│   │   └── health.rs      # Health check models
│   ├── errors.rs          # Error types and handling
│   ├── traits.rs          # Domain traits and interfaces
│   ├── retry/             # Retry logic and policies
│   │   ├── policy.rs      # Retry policies
│   │   └── backoff.rs     # Backoff strategies
│   └── config/            # Configuration models
│       ├── mod.rs         # Config module exports
│       ├── models.rs      # Configuration structures
│       └── validation.rs  # Config validation logic
│
├── services/              # Business logic layer
│   ├── mod.rs            # Service registry and DI container
│   ├── backend_service.rs    # Backend management and selection
│   ├── model_service.rs      # Model aggregation and caching
│   ├── proxy_service.rs      # Request routing and proxying
│   ├── health_service.rs     # Health monitoring
│   └── streaming/        # SSE streaming support
│       ├── mod.rs        # Streaming module exports
│       ├── parser.rs     # SSE event parsing
│       └── handler.rs    # Stream handling logic
│
├── infrastructure/        # External integrations layer
│   ├── mod.rs            # Infrastructure exports
│   ├── backends/         # Backend client implementations
│   │   ├── mod.rs        # Backend trait and factory
│   │   ├── openai.rs     # OpenAI-compatible backend
│   │   ├── ollama.rs     # Ollama backend
│   │   └── vllm.rs       # vLLM backend
│   ├── cache/            # Caching implementations
│   │   ├── mod.rs        # Cache traits
│   │   ├── lru.rs        # LRU cache
│   │   └── ttl.rs        # TTL-based cache
│   ├── config/           # Configuration loading
│   │   ├── loader.rs     # Config file loading
│   │   └── watcher.rs    # Hot-reload support
│   └── http_client.rs    # HTTP client with pooling
│
├── http/                  # HTTP layer
│   ├── mod.rs            # HTTP module exports
│   ├── routes.rs         # Route definitions
│   ├── handlers/         # Request handlers
│   │   ├── chat.rs       # Chat completion endpoints
│   │   ├── models.rs     # Model listing endpoints
│   │   ├── health.rs     # Health check endpoints
│   │   └── admin.rs      # Admin endpoints
│   ├── middleware/       # HTTP middleware
│   │   ├── auth.rs       # Authentication
│   │   ├── logging.rs    # Request logging
│   │   ├── metrics.rs    # Metrics collection
│   │   └── error.rs      # Error handling
│   └── dto/              # Data Transfer Objects
│       ├── request.rs    # Request DTOs
│       └── response.rs   # Response DTOs
│
└── metrics/              # Metrics and monitoring
    ├── mod.rs            # Metrics module exports
    ├── backend.rs        # Backend-specific metrics
    ├── routing.rs        # Routing metrics
    ├── model.rs          # Model service metrics
    └── streaming.rs      # Streaming metrics

tests/
├── integration/          # Integration tests
│   ├── health_test.rs    # Health endpoint tests
│   ├── chat_test.rs      # Chat completion tests
│   ├── models_test.rs    # Model listing tests
│   └── routing_test.rs   # Routing logic tests
├── unit/                 # Unit tests
│   ├── core/            # Core layer tests
│   ├── services/        # Service layer tests
│   └── infrastructure/  # Infrastructure tests
└── common/              # Test utilities
    ├── fixtures.rs      # Test fixtures
    ├── mocks.rs         # Mock implementations
    └── helpers.rs       # Test helper functions

examples/
├── standalone_server.rs  # Minimal server from a config file
├── axum_integration.rs   # Embedding router in an existing Axum app
├── programmatic_config.rs  # Configuration without YAML
└── hot_reload.rs         # Hot-reload and config-change subscriptions

Running Examples

The examples/ directory contains runnable Rust programs that demonstrate the library API. Run any example with:

cargo run --example <name> [-- <args>]

For example:

# Start a standalone server
cargo run --example standalone_server -- --config config.yaml

# Embed the router in a custom Axum app
cargo run --example axum_integration -- --config config.yaml

# Configure programmatically (no YAML file)
OPENAI_API_KEY=sk-... cargo run --example programmatic_config

# Demonstrate hot-reload
cargo run --example hot_reload -- --config config.yaml

See Library Usage for a full guide to the embeddable crate API.

Architecture Overview

Dependency Flow

HTTP Layer → Services Layer → Infrastructure Layer → Core Layer
     ↓            ↓                ↓                    ↑
     └────────────┴────────────────┴────────────────────┘
                        Uses Core Types

Key Design Patterns

  1. Dependency Injection: Services are injected via a container
  2. Repository Pattern: Backend implementations abstract data access
  3. Strategy Pattern: Load balancing strategies are pluggable
  4. Observer Pattern: Health monitoring uses event-driven updates
  5. Circuit-breaker state machine: Standalone state tracking and Admin controls
  6. Retry with Backoff: Handles transient failures

Building and Running

Development Build

# Quick development build
cargo build

# Run with debug output
RUST_LOG=debug ./target/debug/continuum-router

# Run with auto-reload
cargo watch -x run

# Run with specific config
cargo run -- --config dev-config.yaml

Release Build

# Optimized release build (all features enabled by default)
cargo build --release

# Run release build
./target/release/continuum-router

# Build with specific features only (minimal library build)
cargo build --lib --no-default-features --features "metrics,hot-reload"

Cargo Feature Flags

The crate exposes feature flags to enable optional components. The full feature set is enabled by default when building the binary.

Feature Description Default
full Enables all features below yes (via default)
cli CLI binary support via clap yes
admin Admin API endpoints yes
metrics Prometheus metrics endpoint yes
files Files API (/v1/files) yes
health-check Background health monitoring yes
hot-reload Config file watching for live reload yes
webui Embedded WebUI assets yes
gcp-auth Google Cloud auth for Gemini Service Account yes
redis-cache Redis/Valkey backend for the response cache and for distributed rate limiting (deadpool-redis). Not in default/full, but official release binaries and container images compile it; see Deployment for the supported endpoint contract. no (shipped in release builds)
embed Minimal in-process serving surface for embedding the router as a library (e.g. iOS). Enables no extra component and pulls in none of the iOS-hostile optional deps (rusqlite, redis, prometheus, rust-embed, aws-*, clap, notify, rmcp, object_store, gcp_auth). no

When using continuum-router as a library crate, you can select only the features you need to reduce compile times and binary size:

[dependencies]
continuum-router = { git = "...", default-features = false, features = ["metrics", "hot-reload"] }

Build examples:

# Minimal core library (no optional deps)
cargo build --lib --no-default-features

# Library with metrics support
cargo build --lib --no-default-features --features "metrics"

# Library with admin and hot-reload
cargo build --lib --no-default-features --features "admin,hot-reload"

# Minimal in-process embedding surface (e.g. linking into an iOS app)
cargo check --lib --no-default-features --features "embed"
cargo check --lib --no-default-features --features "embed" --target aarch64-apple-ios

# Full binary (default)
cargo build --release

The embed feature is the supported way to link continuum-router as a Rust library and run its HTTP server in-process (for example on iOS, where the app sandbox forbids running the router as a sidecar subprocess). It keeps the dependency graph free of crates that are awkward or impossible to cross-compile for mobile targets. The entrypoint is serve_embedded; verify the excluded dependencies with:

cargo tree --no-default-features --features embed --edges normal --target aarch64-apple-ios

Cross-Compilation

# Install cross
cargo install cross

# Build for Linux x86_64
cross build --release --target x86_64-unknown-linux-gnu

# Build for macOS ARM64
cross build --release --target aarch64-apple-darwin

# Build for Windows
cross build --release --target x86_64-pc-windows-gnu

Testing

Running Tests

# Run all tests
cargo test

# Run with output
cargo test -- --nocapture

# Run specific test
cargo test test_health_check

# Run tests in single thread (for debugging)
cargo test -- --test-threads=1

# Run integration tests only
cargo test --test integration

# Run unit tests only
cargo test --lib

# Run doc tests
cargo test --doc

Writing Tests

Unit Test Example

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_backend_selection() {
        let backends = vec![
            Backend::new("backend1", "http://localhost:8000"),
            Backend::new("backend2", "http://localhost:8001"),
        ];

        let selector = RoundRobinSelector::new(backends);
        let selected = selector.select().unwrap();

        assert_eq!(selected.name(), "backend1");
    }

    #[tokio::test]
    async fn test_async_health_check() {
        let backend = Backend::new("test", "http://localhost:8000");
        let result = backend.health_check().await;

        assert!(result.is_ok());
    }
}

Integration Test Example

Integration tests live in top-level tests/*.rs files, one Cargo test target per file. A .rs file nested in a subdirectory of tests/ is compiled only when a top-level target declares it with mod <dir>;, so a test placed there without that declaration silently never runs. tests/cargo_test_discovery_audit_test.rs fails the build if one is.

// tests/chat_test.rs
use continuum_router::test_utils::TestServer;

#[tokio::test]
async fn test_chat_completion() {
    let server = TestServer::new().await;

    let response = server
        .post("/v1/chat/completions")
        .json(&json!({
            "model": "gpt-5.6-luna",
            "messages": [{"role": "user", "content": "Hello"}]
        }))
        .send()
        .await;

    assert_eq!(response.status(), 200);
}

Test Coverage

# Generate coverage report
cargo tarpaulin --out Html

# Generate coverage with specific threshold
cargo tarpaulin --fail-under 80

# Generate Codecov report
cargo tarpaulin --out Xml

Smart routing classification accuracy

tests/smart_routing_accuracy_test.rs measures how often the rule-based classifier's classify_only output (src/services/smart_routing/router.rs) matches a labelled dataset at tests/data/smart_routing/dataset.json, and reports a confusion matrix per classifier mode. Run the full report:

cargo test --test smart_routing_accuracy_test -- --nocapture

The same command is the CI gate: accuracy_meets_the_regression_gate fails the build when either mode's domain or complexity accuracy drops below the threshold recorded next to DOMAIN_ACCURACY_GATE and COMPLEXITY_ACCURACY_GATE in the test file. On a failing run the report prints automatically, --nocapture is only needed to see it on a passing run.

Two rule-classifier deployment modes are measured (LLM and hybrid modes are out of scope, since they require a live or mocked backend round-trip rather than a fast, deterministic check): rule_default (no primary_language configured) and rule_primary_ko (primary_language: ko, a Korean-deployment classifier). These are the same two configurations tests/classifier_language_parity_test.rs exercises for issue #1567, and they differ only in how a Korean request with no domain-intent signal reads: multilingual under rule_default, general under rule_primary_ko.

Reading the report

Each mode prints overall and per-language accuracy, a domain confusion matrix, an overall complexity confusion matrix, a complexity confusion matrix per language, and a line for every case whose complexity band was wrong. The per-language split and the miss listing were added in #1605, because an aggregate matrix cannot tell you which half of the corpus a confusion cell comes from and the two languages do not fail the same way.

A miss line reads id lang expected -> actual (direction) [signals]. The bracketed list is every signal that fed determine_complexity, including the three domain signals it reads directly (code_blocks, math_notation, tool_definitions), each rendered as name=strength. Use it to trace a wrong band to the signals that produced it rather than guessing: two cases printing the same signal vector cannot be separated by any weight or boundary change, which is how the ceiling documented on COMPLEXITY_ACCURACY_GATE was established.

When you change a weight or a bucket boundary in determine_complexity, justify it by a confusion cell that moves, not by an individual fixture that starts passing. With 59 cases a single cell can hold one or two of them, and a change tuned on that is indistinguishable from overfitting.

Adding a labelled case

Each entry in dataset.json looks like:

{
  "id": "code-fence-en",
  "language": "en",
  "text": "Please implement a function to reverse a string:\n```python\ndef reverse_string(s):\n    pass\n```",
  "expected": {
    "rule_default": { "domain": "code", "complexity": "moderate" },
    "rule_primary_ko": { "domain": "code", "complexity": "moderate" }
  },
  "note": "optional: why this label, or what gap the case is designed to surface"
}
  • id is unique and stable; it is what a future report references when a case's outcome changes.
  • language is "en" or "ko"; the report breaks accuracy down by this field.
  • text is the user message. Use a fenced code block (```) to exercise the code-domain signal, LaTeX delimiters ($...$, \frac, \sum) for the math-notation signal, and Korean or English keywords from the tables in src/services/smart_routing/classifier_language.rs to exercise a specific intent signal.
  • expected must have one entry per mode exercised by the runner (currently rule_default and rule_primary_ko); a case missing either key fails the test with a clear panic rather than being silently skipped.
  • Label by what a human reviewer, reading the DomainTag/ComplexityLevel doc comments in src/services/smart_routing/types.rs, would call the request, not by what the classifier currently outputs for it. Copying the classifier's own output as the label makes accuracy trivially 100% and the gate meaningless; independent labelling is what makes a future regression show up as a real accuracy drop. When the classifier's actual output differs from your label, that is either a genuine limitation worth documenting in note (see creative-poem-ko and code-unmarked-ko in the dataset for examples) or a sign your label needs a second look, not a reason to silently match the output.
  • After editing the dataset, rerun the report command and update the gate constants' doc comments in tests/smart_routing_accuracy_test.rs (the measured percentages, the two failure margins, and what each gate does and does not catch) to match the new measured numbers. Do not raise a gate above what you actually measured.

Code Quality

Formatting

# Format all code
cargo fmt

# Check formatting without changes
cargo fmt -- --check

# Format specific file
cargo fmt -- src/main.rs

Linting

# Run clippy with warnings as errors
cargo clippy -- -D warnings

# Run clippy with pedantic lints
cargo clippy -- -W clippy::pedantic

# Run clippy with auto-fix
cargo clippy --fix

Security Audit

# Check for known vulnerabilities
cargo audit

# Fix vulnerable dependencies
cargo audit fix

# Generate security report
cargo audit --json > audit-report.json

Documentation

# Build documentation
cargo doc

# Build and open documentation
cargo doc --open

# Build documentation with private items
cargo doc --document-private-items

# Check documentation examples
cargo test --doc

Debugging

Debug Logging

# Enable debug logging
RUST_LOG=debug cargo run

# Enable trace logging for specific module
RUST_LOG=continuum_router::services=trace cargo run

# Enable structured logging
RUST_LOG=info RUST_LOG_FORMAT=json cargo run

Using Debugger

VS Code

  1. Create .vscode/launch.json:
{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "lldb",
      "request": "launch",
      "name": "Debug Router",
      "cargo": {
        "args": ["build", "--bin=continuum-router"],
        "filter": {
          "name": "continuum-router",
          "kind": "bin"
        }
      },
      "args": ["--config", "dev-config.yaml"],
      "cwd": "${workspaceFolder}"
    }
  ]
}
  1. Set breakpoints and press F5 to debug

Command Line (lldb)

# Build with debug symbols
cargo build

# Run with lldb
lldb target/debug/continuum-router

# Set breakpoint
(lldb) b src/main.rs:50

# Run program
(lldb) r --config config.yaml

Memory Debugging

# Use valgrind (Linux)
valgrind --leak-check=full ./target/debug/continuum-router

# Use Address Sanitizer
RUSTFLAGS="-Z sanitizer=address" cargo build -Z build-std --target x86_64-unknown-linux-gnu
./target/x86_64-unknown-linux-gnu/debug/continuum-router

Performance Profiling

CPU Profiling

# Generate flame graph
cargo flamegraph -- --config config.yaml

# Profile specific scenario
cargo flamegraph --bin continuum-router -- --benchmark

# Use perf (Linux)
perf record -g ./target/release/continuum-router
perf report

Benchmarking

# Run benchmarks
cargo bench

# Run specific benchmark
cargo bench routing

# Compare benchmark results
cargo bench -- --save-baseline main
git checkout feature-branch
cargo bench -- --baseline main

Writing Benchmarks

// benches/routing_bench.rs
use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn benchmark_routing(c: &mut Criterion) {
    c.bench_function("round_robin_selection", |b| {
        let selector = RoundRobinSelector::new(backends);
        b.iter(|| {
            selector.select(black_box(&request))
        });
    });
}

criterion_group!(benches, benchmark_routing);
criterion_main!(benches);

Memory Profiling

# Use heaptrack (Linux)
heaptrack ./target/release/continuum-router
heaptrack_gui heaptrack.continuum-router.*.gz

# Use Instruments (macOS)
cargo instruments -t Allocations

Contributing

Development Workflow

  1. Fork and Clone 

    git clone https://github.com/YOUR_USERNAME/backend.ai-continuum.git
cd backend.ai-continuum/continuum-router

  2. Create Feature Branch 

    git checkout -b feature/amazing-feature

  3. Make Changes

  4. Write code following Rust conventions
  5. Add tests for new functionality

  6. Update documentation

  7. Test Your Changes 

    cargo test
cargo fmt
cargo clippy -- -D warnings

  8. Commit Changes 

    git add .
git commit -m "feat: Add amazing feature"

  9. Push and Create PR 

    git push origin feature/amazing-feature

Code Standards

  • Follow Rust naming conventions (snake_case for functions, CamelCase for types)
  • Write comprehensive tests (aim for >90% coverage)
  • Document public APIs with doc comments
  • Use cargo fmt for consistent formatting
  • Address all cargo clippy warnings
  • Keep functions small and focused
  • Prefer composition over inheritance
  • Use appropriate error handling (Result/Option)

Commit Message Format

Follow the conventional commits specification:

<type>(<scope>): <subject>

<body>

<footer>

Types: - feat: New feature - fix: Bug fix - docs: Documentation only - style: Code style changes - refactor: Code refactoring - perf: Performance improvements - test: Adding tests - chore: Maintenance tasks

Pull Request Guidelines

  1. Title: Clear and descriptive
  2. Description: Explain what, why, and how
  3. Tests: Include tests for new features
  4. Documentation: Update relevant docs
  5. Breaking Changes: Clearly marked
  6. Issues: Reference related issues

Review Process

  1. Automated checks must pass
  2. At least one maintainer approval
  3. No unresolved conversations
  4. Up-to-date with main branch

Troubleshooting

Common Issues

Compilation Errors

# Clear build cache
cargo clean

# Update dependencies
cargo update

# Check for conflicting features
cargo tree --duplicates

Test Failures

# Run tests verbosely
cargo test -- --nocapture --test-threads=1

# Check for race conditions
cargo test -- --test-threads=1

# Verify test environment
cargo test -- --ignored

Performance Issues

# Profile release build
cargo build --release
perf record ./target/release/continuum-router
perf report

# Check for blocking operations
RUST_LOG=trace cargo run 2>&1 | grep "blocking"

Resources