Development Guide¶

This guide covers the development setup, architecture, testing, and contribution guidelines for the Continuum Router project.

Table of Contents¶

• Prerequisites
• Development Setup
• Project Structure
• Architecture Overview
• Building and Running
• Release Packaging and Code Signing
• Testing
• Code Quality
• Debugging
• Performance Profiling
• Contributing

Prerequisites¶

Required Tools¶

• Rust: 1.97 or later (Install Rust); the root router package and standalone perf/ harness use Rust edition 2024, while this repository pins Rust 1.97.1 in rust-toolchain.toml for reproducible CI and local linting.
• Git: For version control
• C Compiler: gcc, clang, or MSVC (for native dependencies)

Rust edition, MSRV, and formatting¶

The root continuum-router package and standalone perf/ harness declare Rust edition 2024 with rust-version = "1.97". The vendored crates/continuum-protocol package intentionally stays on edition 2021, with the hub workspace MSRV of Rust 1.97, because it mirrors the upstream Continuum Hub protocol crate; Cargo supports this mixed-edition graph, and router CI validates that path dependency separately, including a byte-identity check against its pinned hub revision (see Vendored continuum-protocol crate). rustfmt.toml pins style_edition = "2021" so cargo fmt and editor integrations keep the existing formatting style until a separate formatting-only migration is accepted.

Recommended Tools¶

• cargo-watch: For auto-recompilation during development
• cargo-edit: For managing dependencies
• cargo-audit: For security vulnerability checking
• cargo-flamegraph: For performance profiling
• rust-analyzer: IDE support (VS Code, IntelliJ, vim/neovim)

Install Development Tools¶

# Install essential development tools
cargo install cargo-watch cargo-edit cargo-audit cargo-flamegraph

# Install code coverage tools
cargo install cargo-tarpaulin

# Install benchmarking tools
cargo install cargo-criterion

Development Setup¶

Clone and Setup¶

# Clone the repository
git clone https://github.com/lablup/continuum-router.git
cd continuum-router

# Check that everything compiles
cargo check

# Run initial test suite
cargo test

# Build development version
cargo build

make setup installs the development tools and points Git at the checked-in hooks in .githooks/ (core.hooksPath). The pre-commit hook runs cargo fmt and Clippy and stops the commit when either one changes the tree, so stage the changes and commit again. If core.hooksPath is already set to another directory, make setup leaves it unchanged and prints the command to switch.

IDE Setup¶

VS Code¶

1. Install the rust-analyzer extension
2. Install the CodeLLDB extension for debugging
3. Use the provided .vscode/settings.json:

{
  "rust-analyzer.cargo.features": "all",
  "rust-analyzer.checkOnSave.command": "clippy",
  "editor.formatOnSave": true,
  "[rust]": {
    "editor.defaultFormatter": "rust-lang.rust-analyzer"
  }
}

IntelliJ IDEA / CLion¶

1. Install the Rust plugin
2. Open the project root
3. Configure Cargo settings in Preferences → Build → Cargo

Vim/Neovim¶

1. Install rust-analyzer via Mason or manually
2. Configure LSP client (nvim-lspconfig, coc.nvim, etc.)
3. Example nvim-lspconfig setup:

require('lspconfig').rust_analyzer.setup({
  settings = {
    ["rust-analyzer"] = {
      cargo = { features = "all" },
      checkOnSave = { command = "clippy" }
    }
  }
})

Project Structure¶

Layer Architecture¶

The project follows a clean 4-layer architecture:

Directory Structure¶

src/
├── main.rs                 # Application entry point and CLI
├── lib.rs                  # Library exports and module declarations
│
├── core/                   # Core domain layer (no external dependencies)
│   ├── mod.rs             # Core module exports
│   ├── models/            # Domain models
│   │   ├── backend.rs     # Backend configuration and state
│   │   ├── model.rs       # LLM model representations
│   │   ├── request.rs     # Request/response models
│   │   └── health.rs      # Health check models
│   ├── errors.rs          # Error types and handling
│   ├── traits.rs          # Domain traits and interfaces
│   ├── retry/             # Retry logic and policies
│   │   ├── policy.rs      # Retry policies
│   │   └── backoff.rs     # Backoff strategies
│   └── config/            # Configuration models
│       ├── mod.rs         # Config module exports
│       ├── models.rs      # Configuration structures
│       └── validation.rs  # Config validation logic
│
├── services/              # Business logic layer
│   ├── mod.rs            # Module exports
│   ├── model_service.rs      # Model aggregation and caching
│   ├── proxy_service.rs      # Request routing and proxying
│   ├── health_service.rs     # Health monitoring
│   └── streaming/        # SSE streaming support
│       ├── mod.rs        # Streaming module exports
│       ├── parser.rs     # SSE event parsing
│       └── handler.rs    # Stream handling logic
│
├── infrastructure/        # External integrations layer
│   ├── mod.rs            # Infrastructure exports
│   ├── backends/         # Backend client implementations
│   │   ├── mod.rs        # Backend trait and factory
│   │   ├── openai.rs     # OpenAI-compatible backend
│   │   ├── ollama.rs     # Ollama backend
│   │   └── vllm.rs       # vLLM backend
│   ├── cache/            # Caching implementations
│   │   ├── mod.rs        # Cache traits
│   │   ├── lru.rs        # LRU cache
│   │   └── ttl.rs        # TTL-based cache
│   ├── config/           # Configuration loading
│   │   ├── loader.rs     # Config file loading
│   │   └── watcher.rs    # Hot-reload support
│   └── common/http_client.rs  # HTTP client factory with pooling
│
├── http/                  # HTTP layer
│   ├── mod.rs            # HTTP module exports
│   ├── routes.rs         # Route definitions
│   ├── handlers/         # Request handlers
│   │   ├── chat.rs       # Chat completion endpoints
│   │   ├── models.rs     # Model listing endpoints
│   │   ├── health.rs     # Health check endpoints
│   │   └── admin.rs      # Admin endpoints
│   ├── middleware/       # HTTP middleware
│   │   ├── auth.rs       # Authentication
│   │   ├── logging.rs    # Request logging
│   │   ├── metrics.rs    # Metrics collection
│   │   └── error.rs      # Error handling
│   └── dto/              # Data Transfer Objects
│       ├── request.rs    # Request DTOs
│       └── response.rs   # Response DTOs
│
└── metrics/              # Metrics and monitoring
    ├── mod.rs            # Metrics module exports
    ├── backend.rs        # Backend-specific metrics
    ├── routing.rs        # Routing metrics
    ├── model.rs          # Model service metrics
    └── streaming.rs      # Streaming metrics

tests/
├── rate_limiting_test.rs # A top-level target: CI selects it by name or it
│                         # reads process-global state
├── streaming/            # A group target (issue #1620)
│   ├── main.rs           # `pub mod <stem>;` per member, helpers via #[path]
│   ├── streaming_test.rs # Member: a former top-level file, unchanged
│   └── ...
├── server/, routing/, guardrails/, ...  # The other group targets
├── tool_calling/         # Tool-calling suite, its own target (main.rs)
├── common/               # Shared helpers (mock servers, fixtures)
├── *_support/, *_common/ # Other shared helpers, declared by the targets
│                         # that use them
└── fixtures/             # Test data

examples/
├── standalone_server.rs  # Minimal server from a config file
├── axum_integration.rs   # Embedding router in an existing Axum app
├── programmatic_config.rs  # Configuration without YAML
└── hot_reload.rs         # Hot-reload and config-change subscriptions

Running Examples¶

The examples/ directory contains runnable Rust programs that demonstrate the library API. Run any example with:

cargo run --example <name> [-- <args>]

For example:

# Start a standalone server
cargo run --example standalone_server -- --config config.yaml

# Embed the router in a custom Axum app
cargo run --example axum_integration -- --config config.yaml

# Configure programmatically (no YAML file)
OPENAI_API_KEY=sk-... cargo run --example programmatic_config

# Demonstrate hot-reload
cargo run --example hot_reload -- --config config.yaml

See Library Usage for a full guide to the embeddable crate API.

Architecture Overview¶

Dependency Flow¶

HTTP Layer → Services Layer → Infrastructure Layer → Core Layer
     ↓            ↓                ↓                    ↑
     └────────────┴────────────────┴────────────────────┘
                        Uses Core Types

Key Design Patterns¶

1. Constructor Injection: Services are passed as constructor arguments; there is no runtime DI container
2. Repository Pattern: Backend implementations abstract data access
3. Strategy Pattern: Load balancing strategies are pluggable
4. Observer Pattern: Health monitoring uses event-driven updates
5. Circuit-breaker state machine: Standalone state tracking and Admin controls
6. Retry with Backoff: Handles transient failures

Building and Running¶

Development Build¶

# Quick development build
cargo build

# Run with debug output
RUST_LOG=debug ./target/debug/continuum-router

# Run with auto-reload
cargo watch -x run

# Run with specific config
cargo run -- --config dev-config.yaml

Release Build¶

# Optimized release build (all features enabled by default)
cargo build --release

# Run release build
./target/release/continuum-router

# Build with specific features only (minimal library build)
cargo build --lib --no-default-features --features "metrics,hot-reload"

The committed .cargo/config.toml sets no target-cpu, so every build above targets the portable baseline for its architecture, the same as what release binaries ship. To opt into native CPU features for a local build, set RUSTFLAGS for that invocation rather than editing the committed file: RUSTFLAGS="-C target-cpu=native" cargo build --release. A user-level ~/.cargo/config.toml works too, and neither is ever committed or shared through a CI cache: cached proc-macro dylibs built for one CPU generation crash rustc with SIGILL when restored onto a different one on GitHub's heterogeneous hosted runners.

Cargo Feature Flags¶

The crate exposes feature flags to enable optional components. The full feature set is enabled by default when building the binary.

When using continuum-router as a library crate, you can select only the features you need to reduce compile times and binary size:

[dependencies]
continuum-router = { git = "...", default-features = false, features = ["metrics", "hot-reload"] }

Build examples:

# Minimal core library (no optional deps)
cargo build --lib --no-default-features

# Library with metrics support
cargo build --lib --no-default-features --features "metrics"

# Library with admin and hot-reload
cargo build --lib --no-default-features --features "admin,hot-reload"

# Minimal in-process embedding surface (e.g. linking into an iOS app)
cargo check --lib --no-default-features --features "embed"
cargo check --lib --no-default-features --features "embed" --target aarch64-apple-ios

# Full binary (default)
cargo build --release

The embed feature is the supported way to link continuum-router as a Rust library and run its HTTP server in-process (for example on iOS, where the app sandbox forbids running the router as a sidecar subprocess). It keeps the dependency graph free of crates that are awkward or impossible to cross-compile for mobile targets. The entrypoint is serve_embedded; verify the excluded dependencies with:

cargo tree --no-default-features --features embed --edges normal --target aarch64-apple-ios

Cross-Compilation¶

# Install cross
cargo install cross

# Build for Linux x86_64
cross build --release --target x86_64-unknown-linux-gnu

# Build for macOS ARM64
cross build --release --target aarch64-apple-darwin

# Build for Windows
cross build --release --target x86_64-pc-windows-gnu

CI runner usage and build reuse¶

Automatic self-hosted CI runs on [self-hosted, macOS], supporting both Intel and Apple Silicon machines:

• Concurrency groups: Two repository-wide concurrency groups bound simultaneous jobs across pull requests, dependency floats, and documentation builds:
  • continuum-router-self-hosted-1: admits the default graph, isolated graphs, and weekly dependency float.
  • continuum-router-self-hosted-2: admits change detection, the release graph, deployment validation, IDE drift checks, and documentation builds.
  • Each group uses queue: max with cancel-in-progress: false, preserving up to 100 waiting jobs in FIFO order instead of replacing another PR's pending check (while workflow-level concurrency still cancels older runs of the same PR on a new push). This bounds automatic jobs in this repository, not other organization repositories or manually dispatched Dynamo E2E jobs with custom runner labels; see GitHub's concurrency documentation.
• Two-build test execution: Both the CI workflow and make ci-local run tests from two library builds:
  • Default features (cargo test --tests in ci-default), retained because six tests gated on not(feature = "control-plane") or not(feature = "bedrock-sigv4") exist only there.
  • All features (cargo test --all-features --no-fail-fast --lib --tests in ci-release-graph), where --no-fail-fast runs the remaining binaries after a failure.
  • Different feature graphs, normal builds versus test builds, and clippy versus executable tests still require separate compilations (a repeated Compiling continuum-router line can name a new integration binary rather than rebuilding src/lib.rs). Verbose Fresh output is omitted, and cargo-deny reports duplicate-version warnings without long inclusion graphs.
• Local CI helpers:
  • scripts/local-ci.sh --only <test target> reruns a single exact target under every feature (exiting 2 if no step matches or if every selected step was skipped with No step ran, and reporting the skipped count on the last line when steps are skipped).
  • make ci-local-gated skips the feature-graph steps that the hosted gate would skip for the branch (running everything when there is no merge base with origin/main, when the work tree is dirty, or when the gate errors or widens), while ungated make ci-local remains the merge gate.
• Toolchains, caching, and debug symbols: Rust jobs install the numeric channel from rust-toolchain.toml via .github/actions/rust-toolchain, and native self-hosted artifacts stay in $HOME/actions-cargo-target/continuum-router (outside checkout cleanup, with an 80 GB ceiling, kept separate per Intel/Apple Silicon host). CI and local mirrors default to CARGO_PROFILE_DEV_DEBUG=0 and CARGO_PROFILE_TEST_DEBUG=0 to reduce debug artifacts and link time while keeping assertions (set either to 2 locally when symbols are needed); the performance workflow retains debug symbols for crash diagnostics and uses the shipped release profile for capacity measurements.

update_homebrew_formula.yml_ and launchpad_ppa.yml_ are inactive files, not scheduled workflows. GitHub-hosted jobs remain until their required VM capabilities are provisioned; the efficiency changes add no new hosted jobs.

Release Packaging and Code Signing¶

The release workflow treats signing as part of artifact construction, not as optional post-processing. The four Linux matrix entries remain unsigned and unchanged. The macOS and Windows paths fail before archive, checksum, or release upload when their signing contract is not satisfied.

Platform flow¶

• macOS ARM64 runs on self-hosted-macos-26-arm64. .github/actions/macos-signing-setup finds a capable Homebrew OpenSSL through the known /opt/homebrew and /usr/local keg and brew paths before falling back to PATH, passes the .p12 password through the environment, extracts a Developer ID Application certificate for keychain-free rcodesign, and pins SHA-256 digests for both the rcodesign archive and executable. The archive is verified before extraction, and an extracted or cached executable is verified before execution or reuse. .github/actions/macos-sign-notarize-binary signs both continuum-router and continuum-kv-listener with the hardened runtime, verifies their authority and identifiers, creates the zip, and requires notarytool to report status: Accepted. Any other result fetches the notarization log and fails. An always() cleanup step removes the temporary .p12 and PEM immediately afterward. Apple does not staple bare command-line executables or zip containers, so Gatekeeper retrieves the ticket online.
• Windows x86_64 builds and signs on windows-on-macmini02-x64. Both executables are copied to a clean staging directory, signed through scripts/sign-windows.ps1, and verified again with signtool verify /pa /all /v before the zip is created. LICENSE and NOTICE are included beside the executables.
• Windows ARM64 builds on the GitHub-hosted windows-11-arm runner. The locked ring and aws-lc-sys dependencies require Clang-compatible ARM64 assembly support, so .github/actions/setup-clang-windows-arm64 verifies clang and clang-cl before Cargo runs. The builder uploads only the two unsigned PE files plus LICENSE and NOTICE as a one-day workflow artifact. The x86_64 signing runner checks both PE machine fields for ARM64, signs and fail-closed verifies both files, then creates continuum-router-windows-aarch64.zip, its checksum, and the release upload. No unsigned ARM64 archive is created or uploaded. The signer checks out only scripts/sign-windows.ps1, into its own signing-checkout/ directory: it shares this runner's workspace with the x86_64 build, and a sparse checkout at the workspace root once left the next build job without the repository's files.

Both Windows targets use the static MSVC CRT. Authenticode hashes a PE file without executing it, so the x86_64 signing host can sign ARM64 outputs. Every job that consumes the completed build, including release notifications and container publication, also depends on the ARM64 signing job and cannot report success while that artifact is unsigned or missing.

Safe signing test mode¶

A manual workflow_dispatch can set the boolean signing_test input to true to exercise the production signing path without creating or changing a release. The default is false, so existing release events and manual production dispatches retain their previous behavior. Test mode runs all seven build matrix entries, performs real macOS signing and notarization, signs and verifies Windows x86_64, and completes the hosted Windows ARM64 handoff on the restricted signer.

Completed archives and checksums are retained only as GitHub Actions artifacts for three days. The Windows ARM64 final artifact is named signing-test-final-signed-aarch64-pc-windows-msvc, which is distinct from the one-day unsigned-aarch64-pc-windows-msvc handoff. The workflow does not invoke either GitHub Release upload, create a tag or release, publish either container image, mirror to Harbor, notify Teams, or update Homebrew while signing_test is true. Its explicit Signing test (no publication) run title also makes the downstream Debian packaging workflow skip the completion event instead of modifying an existing release, while the staging workflow already accepts only release completion events.

The top-level workflow name is Release Packaging, migrated from Release together with both workflow_run consumers. GitHub evaluates a workflow_run consumer from the default branch, so a branch-only condition in debian_build.yml cannot protect the first pre-merge test. The distinct name closes that rollout gap: while this change is still on a pull request, default-branch consumers listen only for the old Release name and cannot be triggered by the branch's Release Packaging run; after merge, both consumers listen for Release Packaging, and the run-title and event-type guards provide the steady-state protection. The rename changes orchestration identity only; release events, manual production dispatches, asset uploads, containers, Debian packaging, and staging keep their normal behavior.

Dispatch the test against the revision under review and leave release_tag empty:

gh workflow run release.yml --ref <branch> -f signing_test=true

Packaging environment¶

Configure these names in the protected packaging environment. Never commit or print their values.

The Windows private key is not exported to GitHub. The self-hosted signing runner must already have the Windows SDK, the KMS CSP, access to the configured certificate chain, and authorization to use the key. The helper does not echo configured paths or the key container; missing files are reported by variable name, and every deployment-specific value must be stored as a masked environment secret.

Validation and rollout¶

Static checks that do not need signing credentials are safe to run on a development machine:

python3 scripts/check_release_signing.py
python3 scripts/test_check_release_signing.py
python3 scripts/check_action_pins.py
actionlint -shellcheck= .github/workflows/release.yml
cargo tree --locked --target aarch64-pc-windows-msvc --features control-plane,appproxy-router,hot-reload,redis-cache,otel,dynamo,dynamo-kv,dynamo-event-bus
cargo tree --locked --manifest-path crates/continuum-kv-listener/Cargo.toml --target aarch64-pc-windows-msvc

cargo tree verifies target-specific dependency resolution only. A rollout is complete only after a controlled signing_test dispatch runs on the configured runners and all of the following are checked:

1. Both Windows archives contain continuum-router.exe, continuum-kv-listener.exe, LICENSE, and NOTICE, and their .sha256 files validate.
2. On clean x86_64 and ARM64 Windows machines, both executables report a valid expected publisher through Get-AuthenticodeSignature and signtool verify /pa /all /v.
3. On a clean networked macOS machine, both extracted executables show a Developer ID Application authority, the hardened runtime flag, and pass Gatekeeper without an unidentified-developer warning.
4. The macOS notarization log reports Accepted, the final Windows ARM64 Actions artifact is distinct from the unsigned handoff, and every release upload, container publication, Harbor mirror, notification, and Homebrew path was skipped.

Do not remove old credentials or change runner provisioning until that signing-test evidence is retained. Secret creation, key access, and clean-machine verification are operational steps and cannot be proven by the repository's static tests.

Testing¶

Running Tests¶

# Run all tests
cargo test

# Run with output
cargo test -- --nocapture

# Run specific test
cargo test test_health_check

# Run tests in single thread (for debugging)
cargo test -- --test-threads=1

# Run integration tests only
cargo test --test '*'

# Run unit tests only
cargo test --lib

# Run doc tests
cargo test --doc

Writing Tests¶

Unit Test Example¶

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_backend_selection() {
        let backends = vec![
            Backend::new("backend1", "http://localhost:8000"),
            Backend::new("backend2", "http://localhost:8001"),
        ];

        let selector = RoundRobinSelector::new(backends);
        let selected = selector.select().unwrap();

        assert_eq!(selected.name(), "backend1");
    }

    #[tokio::test]
    async fn test_async_health_check() {
        let backend = Backend::new("test", "http://localhost:8000");
        let result = backend.health_check().await;

        assert!(result.is_ok());
    }
}

Integration Test Example¶

Integration test structure and CI discovery rules:

• Target discovery and domain groups: Cargo discovers integration test targets as top-level tests/*.rs files and as tests/<group>/main.rs. Most integration tests live in a domain group (such as tests/streaming/ or tests/guardrails/) whose main.rs declares each member with pub mod <stem>; so the group links once; see "Where a new test file may live" in CLAUDE.md.
  • Run one member with cargo test --test <group> <stem>:: (substring match) or a single exact test with cargo test --test <group> -- --exact <stem>::<test>.
  • Files that CI selects by name or that read process-global state remain top-level tests/*.rs targets.
• Shared state isolation: State keyed by a test-chosen ID (an upstream response, file, or conversation ID) is shared across all members of a group binary unless the store is scoped to its own AppState (as the Responses session store is). Prefix test-chosen IDs with the file name to prevent collisions.
• Unreferenced file audit: A .rs file under tests/ that no target declares is never compiled; tests/cargo_test_discovery_audit_test.rs fails the build if an undeclared file exists.
• Feature-gated test audit: Tests gated on a non-default feature (#![cfg(feature = ...)] files, gated #[test] functions, or gated inline mod blocks) compile to nothing in cargo test --tests and run in the all-features batch step (cargo test --all-features --no-fail-fast --lib --tests) of both .github/workflows/ci.yml and scripts/local-ci.sh:
  • The discovery audit fails if neither that batch step nor an explicit cargo test --features <F> --test <target> line covers a gate in both files.
  • In ci.yml, the batch command must be the entire single-line run: of a step with only name and run (no continue-on-error, and if: matching the pinned condition); in scripts/local-ci.sh, it must be a single step "<name>" cargo test ... line with a unique name printed by scripts/local-ci.sh --list. Do not wrap, split, condition, or filter that line.
• PR change detection: On a pull request, the hosted gate schedules ci-release-graph for a changed test target only when it contains a feature-gated test; if the scanner cannot decide (such as a feature name passed via macro argument, an unreadable #[path]/include! module, or many unterminated quotes), it runs the full graph.

// tests/chat_test.rs
use continuum_router::test_utils::TestServer;

#[tokio::test]
async fn test_chat_completion() {
    let server = TestServer::new().await;

    let response = server
        .post("/v1/chat/completions")
        .json(&json!({
            "model": "gpt-5.6-luna",
            "messages": [{"role": "user", "content": "Hello"}]
        }))
        .send()
        .await;

    assert_eq!(response.status(), 200);
}

Test Coverage¶

# Generate coverage report
cargo tarpaulin --out Html

# Generate coverage with specific threshold
cargo tarpaulin --fail-under 80

# Generate Codecov report
cargo tarpaulin --out Xml

Smart routing classification accuracy¶

tests/smart_routing_accuracy_test.rs measures how often the rule-based classifier's classify_only output (src/services/smart_routing/router.rs) matches a labelled dataset at tests/data/smart_routing/dataset.json, and reports a confusion matrix per classifier mode. Run the full report:

cargo test --test smart_routing_accuracy_test -- --nocapture

The same command is the CI gate: accuracy_meets_the_regression_gate fails the build when either mode's domain or complexity accuracy drops below the threshold recorded next to DOMAIN_ACCURACY_GATE and COMPLEXITY_ACCURACY_GATE in the test file. On a failing run the report prints automatically, --nocapture is only needed to see it on a passing run.

Two rule-classifier deployment modes are measured (LLM and hybrid modes are out of scope, since they require a live or mocked backend round-trip rather than a fast, deterministic check): rule_default (no primary_language configured) and rule_primary_ko (primary_language: ko, a Korean-deployment classifier). These are the same two configurations tests/routing/classifier_language_parity_test.rs exercises, and they differ only in how a Korean request with no domain-intent signal reads: multilingual under rule_default, general under rule_primary_ko.

A third mode, system_one, replays the same cases through classifier.method: system_one against the System One fixture server in tests/systemone_support/:

• Fidelity check (100% gate): The fixture is scripted from the rule_primary_ko labels (where a Korean greeting is general, matching the battery's multilingual definition of "translation or explicitly multilingual tasks") and keyed by the exact state the classifier sends for each case. Because this mode tests the classifier's fidelity to the decision model's answers across every case (including the Korean half) rather than the model's own accuracy, its gate is 100% on both axes: any miss means a score was rounded to the wrong level, a domain option was mapped to the wrong tag, or the state drifted from the scripted fixture.
• Live model calibration: The decision model's own accuracy is measured by make calibrate-systemone, which replays the same corpora through each consumer's state builder and question battery against a live /v1/systemone endpoint and derives thresholds from the result; see Decision Models.

Reading the report¶

Each mode prints overall and per-language accuracy, a domain confusion matrix, an overall complexity confusion matrix, a complexity confusion matrix per language, and a line for every case whose complexity band was wrong. An aggregate matrix cannot tell you which half of the corpus a confusion cell comes from, and the two languages do not fail the same way.

A miss line reads id lang expected -> actual (direction) [signals]. The bracketed list is every signal that fed determine_complexity, including the three domain signals it reads directly (code_blocks, math_notation, tool_definitions), each rendered as name=strength. Use it to trace a wrong band to the signals that produced it rather than guessing: two cases printing the same signal vector cannot be separated by any weight or boundary change, which is how the ceiling documented on COMPLEXITY_ACCURACY_GATE was established.

When you change a weight or a bucket boundary in determine_complexity, justify it by a confusion cell that moves, not by an individual fixture that starts passing. With 82 cases a single cell can hold one or two of them, and a change tuned on that is indistinguishable from overfitting.

Adding a labelled case¶

Each entry in dataset.json looks like:

{
  "id": "code-fence-en",
  "language": "en",
  "text": "Please implement a function to reverse a string:\n```python\ndef reverse_string(s):\n    pass\n```",
  "expected": {
    "rule_default": { "domain": "code", "complexity": "moderate" },
    "rule_primary_ko": { "domain": "code", "complexity": "moderate" }
  },
  "note": "optional: why this label, or what gap the case is designed to surface"
}

• id is unique and stable; it is what a future report references when a case's outcome changes.
• language is "en" or "ko", the language of text (the classified turn) even when context is in another language; the report breaks accuracy down by this field.
• context is optional: the earlier messages of a multi-turn conversation, as chat message objects ({"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}), placed in order before text. Omit it for a single-turn case. Use it when the behavior under test depends on history, such as the proof floor reading only the latest user turn; reasoning-proof-explain-ko-after-en is an example. The system_one mode builds its fixture state from the same payload, so a case with context needs no extra wiring.
• text is the latest user message, the request being classified. Use a fenced code block (```) to exercise the code-domain signal, LaTeX delimiters ($...$, \frac, \sum) for the math-notation signal, and Korean or English keywords from the tables in src/services/smart_routing/classifier_language.rs to exercise a specific intent signal.
• expected must have one entry per mode exercised by the runner (currently rule_default and rule_primary_ko); a case missing either key fails the test with a clear panic rather than being silently skipped.
• Label by what a human reviewer reading the DomainTag/ComplexityLevel doc comments in src/services/smart_routing/types.rs would call the request, not by what the classifier currently outputs:
  • Copying the classifier's own output makes accuracy trivially 100% and renders the regression gate meaningless.
  • When the classifier's output differs from your label, either document the genuine limitation in note (see creative-poem-ko and code-unmarked-ko in the dataset) or re-check your label rather than silently matching the output.
• A case's note is printed under its miss line. Any expert-labelled case that misses must carry a note, or the gate fails naming that case: because the rule classifier cannot reach expert for every request the taxonomy calls expert (see the ceiling on COMPLEXITY_ACCURACY_GATE and the Complexity Levels section of the advanced configuration guide), the note must state which part of that ceiling applies.
• After editing the dataset, rerun the report command and update the gate constants' doc comments in tests/smart_routing_accuracy_test.rs (measured percentages, failure margins, and coverage notes) to match the new measurements. Never raise a gate above what you actually measured.

System One fixture server and evaluation corpora¶

tests/systemone_support/ provides the shared test harness for System One decision-model seams:

• Harness components: Includes a wiremock /v1/systemone fixture driven by tests/systemone_support/answers.yaml, fault injection (429, 529, 422, timeouts, and malformed bodies), calibration metrics (accuracy, Brier, 10-bin ECE, AUROC, confusion matrix), and a replay runner.
• Importing in tests: Load it from a top-level integration test with mod systemone_support;, or from a domain group member with use crate::systemone_support; after declaring #[path = "../systemone_support/mod.rs"] once in the group's main.rs (see "Where a new test file may live" in CLAUDE.md). Wire types are kept test-local so src/services/systemone/ is tested against an independent fixture definition.
• Evaluation corpora: Labelled corpora live under tests/fixtures/systemone/ (guardrail.jsonl, routing.jsonl, tool_calls.jsonl, and batteries.yaml), with row schemas, sources, licences, and unsafe-intent handling documented in tests/fixtures/systemone/README.md.
  • tests/systemone_corpus_test.rs fails the build if a label is invalid for its battery, a licence is missing, or row counts fall below acceptance floors (200 rows per corpus, including 40 Korean and 20 non-Latin-script guardrail rows).
  • tests/systemone/systemone_cookbook_regression_test.rs verifies that the runner and metric calculations reproduce the TypeSafe cookbook's published verdicts from cookbook_cache.yaml.

For a quick single-corpus check against a real endpoint, replay it with the #[ignore]d runners, which print the JSON report (the full calibration procedure, which replays every consumer and derives thresholds, is make calibrate-systemone; see Decision Models):

SYSTEMONE_ENDPOINT=http://127.0.0.1:8000/v1/systemone \
SYSTEMONE_API_KEY_ENV=LAYA_SERVE_API_KEY \
cargo test --test systemone_corpus_test -- --ignored run_routing --nocapture

SYSTEMONE_API_KEY_ENV names the variable holding the key; SYSTEMONE_MODEL overrides the requested model (default jev-latest). deploy/docker-compose.systemone.yml brings up a self-hosted laya-serve on loopback for a local run.

Vendored continuum-protocol crate¶

crates/continuum-protocol/src and tests are a byte-identical copy of a single Continuum Hub revision, recorded on the provenance line in crates/continuum-protocol/Cargo.toml (# Vendored verbatim from lablup/continuum-hub crates/continuum-protocol at rev <sha>.):

• Manifest and drift check: crates/continuum-protocol/UPSTREAM.sha256 records the SHA-256 of every Hub file under src/ and tests/ at that revision (plus the digest of the Hub's crate Cargo.toml). scripts/check-vendored-protocol.sh fails (naming each offending path) when the manifest revision differs from the provenance line, when any file is modified, missing, or added, or when a LOCAL-AHEAD entry is malformed or stale.
• Build-input confinement: Because only src/ and tests/ are compared, the script also rejects any extra crate-root entry (such as build.rs), any crate Cargo.toml section outside the allowlist ([package], [features], and registry dependency tables on default targets), any cargo metadata target outside src/ and tests/, and any router Cargo.toml that resolves continuum-protocol from anywhere other than path = "crates/continuum-protocol".
• Execution: Requires no network or Hub credentials, runs first in the release-graph job of CI and make ci-local, and is self-tested across all failure modes by scripts/test-check-vendored-protocol.sh.

# Offline check
bash scripts/check-vendored-protocol.sh

# What CI and make ci-local run: also fail a manifest edit that kept its rev line
bash scripts/check-vendored-protocol.sh --base "$(git merge-base origin/main HEAD)"

# Also regenerate the manifest from a hub clone and compare it with the committed one
CONTINUUM_HUB_DIR=../continuum-hub bash scripts/check-vendored-protocol.sh

# Re-vendor from a hub revision: replaces src/ and tests/, rewrites UPSTREAM.sha256
bash scripts/refresh-vendored-protocol-manifest.sh --hub ../continuum-hub --rev <sha> --sync

--base exists because the natural reaction to "digest differs" is to update the digest: a manifest whose hashes changed while its # rev: line did not can only be a vendored file edited together with its manifest line. Export CONTINUUM_HUB_DIR for make ci-local whenever a hub clone is at hand; that run is the merge gate.

The refresh script never edits Cargo.toml, and in a git work tree it refuses to --sync over uncommitted or untracked changes under src/ or tests/ unless given --force. After a re-vendor, put the provenance line it prints into crates/continuum-protocol/Cargo.toml, carry over any dependency change in git -C ../continuum-hub show <sha>:crates/continuum-protocol/Cargo.toml (the script notes when that file changed), and run the check.

To change a vendored file ahead of the hub, file the hub issue first, then add <path> lablup/continuum-hub#<N> <one-line reason> to crates/continuum-protocol/LOCAL-AHEAD in the same PR as the change, and leave UPSTREAM.sha256 alone. Entries are file paths under src/ or tests/, never directories or globs. The check fails on an entry whose file already matches the pinned revision, so drop it when a re-vendor picks up the hub change.

Code Quality¶

Formatting¶

# Format all code
cargo fmt

# Check formatting without changes
cargo fmt -- --check

# Format specific file
cargo fmt -- src/main.rs

Linting¶

# Run clippy with warnings as errors
cargo clippy -- -D warnings

# Run clippy with pedantic lints
cargo clippy -- -W clippy::pedantic

# Run clippy with auto-fix
cargo clippy --fix

Security Audit¶

# Check for known vulnerabilities
cargo audit

# Fix vulnerable dependencies
cargo audit fix

# Generate security report
cargo audit --json > audit-report.json

Documentation¶

# Build documentation
cargo doc

# Build and open documentation
cargo doc --open

# Build documentation with private items
cargo doc --document-private-items

# Check documentation examples
cargo test --doc

Debugging¶

Debug Logging¶

# Enable debug logging
RUST_LOG=debug cargo run

# Enable trace logging for specific module
RUST_LOG=continuum_router::services=trace cargo run

# Enable structured logging
RUST_LOG=info RUST_LOG_FORMAT=json cargo run

Using Debugger¶

VS Code¶

1. Create .vscode/launch.json:

{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "lldb",
      "request": "launch",
      "name": "Debug Router",
      "cargo": {
        "args": ["build", "--bin=continuum-router"],
        "filter": {
          "name": "continuum-router",
          "kind": "bin"
        }
      },
      "args": ["--config", "dev-config.yaml"],
      "cwd": "${workspaceFolder}"
    }
  ]
}

1. Set breakpoints and press F5 to debug

Command Line (lldb)¶

# Build with debug symbols
cargo build

# Run with lldb
lldb target/debug/continuum-router

# Set breakpoint
(lldb) b src/main.rs:50

# Run program
(lldb) r --config config.yaml

Memory Debugging¶

# Use valgrind (Linux)
valgrind --leak-check=full ./target/debug/continuum-router

# Use Address Sanitizer
RUSTFLAGS="-Z sanitizer=address" cargo build -Z build-std --target x86_64-unknown-linux-gnu
./target/x86_64-unknown-linux-gnu/debug/continuum-router

Performance Profiling¶

CPU Profiling¶

# Generate flame graph
cargo flamegraph -- --config config.yaml

# Profile specific scenario
cargo flamegraph --bin continuum-router -- --benchmark

# Use perf (Linux)
perf record -g ./target/release/continuum-router
perf report

Benchmarking¶

# Run benchmarks
cargo bench

# Run specific benchmark
cargo bench routing

# Compare benchmark results
cargo bench -- --save-baseline main
git checkout feature-branch
cargo bench -- --baseline main

Writing Benchmarks¶

// benches/routing_bench.rs
use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn benchmark_routing(c: &mut Criterion) {
    c.bench_function("round_robin_selection", |b| {
        let selector = RoundRobinSelector::new(backends);
        b.iter(|| {
            selector.select(black_box(&request))
        });
    });
}

criterion_group!(benches, benchmark_routing);
criterion_main!(benches);

Memory Profiling¶

# Use heaptrack (Linux)
heaptrack ./target/release/continuum-router
heaptrack_gui heaptrack.continuum-router.*.gz

# Use Instruments (macOS)
cargo instruments -t Allocations

Contributing¶

Development Workflow¶

1. Fork and Clone

   git clone https://github.com/YOUR_USERNAME/continuum-router.git
   cd continuum-router
   

2. Create Feature Branch

   git checkout -b feature/amazing-feature
   

3. Make Changes

4. Write code following Rust conventions
5. Add tests for new functionality

6. Update documentation

7. Test Your Changes

   cargo test
   cargo fmt
   cargo clippy -- -D warnings
   

8. Commit Changes

   git add .
   git commit -m "feat: Add amazing feature"
   

9. Push and Create PR

   git push origin feature/amazing-feature
   

Code Standards¶

• Follow Rust naming conventions (snake_case for functions, CamelCase for types)
• Write comprehensive tests (aim for >90% coverage)
• Document public APIs with doc comments
• Use cargo fmt for consistent formatting
• Address all cargo clippy warnings
• Keep functions small and focused
• Prefer composition over inheritance
• Use appropriate error handling (Result/Option)

Commit Message Format¶

Follow the conventional commits specification:

<type>(<scope>): <subject>

<body>

<footer>

Types: - feat: New feature - fix: Bug fix - docs: Documentation only - style: Code style changes - refactor: Code refactoring - perf: Performance improvements - test: Adding tests - chore: Maintenance tasks

Pull Request Guidelines¶

1. Title: Clear and descriptive
2. Description: Explain what, why, and how
3. Tests: Include tests for new features
4. Documentation: Update relevant docs
5. Breaking Changes: Clearly marked
6. Issues: Reference related issues

Review Process¶

1. Automated checks must pass
2. At least one maintainer approval
3. No unresolved conversations
4. Up-to-date with main branch

Troubleshooting¶

Common Issues¶

Compilation Errors¶

# Clear build cache
cargo clean

# Update dependencies
cargo update

# Check for conflicting features
cargo tree --duplicates

Test Failures¶

# Run tests verbosely
cargo test -- --nocapture --test-threads=1

# Check for race conditions
cargo test -- --test-threads=1

# Verify test environment
cargo test -- --ignored

Performance Issues¶

# Profile release build
cargo build --release
perf record ./target/release/continuum-router
perf report

# Check for blocking operations
RUST_LOG=trace cargo run 2>&1 | grep "blocking"

Resources¶

• Rust Book
• Rust API Guidelines
• Tokio Documentation
• Axum Documentation
• Project Issues
• Project Discussions