Deployment Guide¶

This guide covers deployment surfaces that exist in the repository: release binaries, official container images, Docker Compose, maintained Kubernetes/Kustomize assets, the Helm chart, and custom systemd units. The Debian package does not install a systemd unit.

Before deployment¶

Create and validate a configuration:

continuum-router config generate --template production-ha --output config.yaml
# Set required environment variables referenced by the template.
continuum-router config validate config.yaml

For a smaller starting point, use --template minimal. Available templates are minimal, development, multi-provider, production-ha, api-gateway, kv-cache-optimized, smart-routing, disaggregated, and cost-optimized.

Bind to loopback unless an external listener is required. Enable API and Admin authentication before exposing the service to an untrusted network. The router serves plain HTTP; terminate TLS in a trusted reverse proxy, ingress controller, or load balancer.

Release feature set¶

A default source build uses Cargo's full feature set. Official release binaries and official container images additionally compile control-plane and appproxy-router; both remain runtime opt-in. appproxy-legacy, bedrock-sigv4, redis-cache, and s3-cache remain source-build opt-ins unless a release workflow explicitly adds them.

Runtime configuration cannot enable code that was not compiled. Check /admin/capabilities with Admin credentials when diagnosing a feature-gated deployment.

Docker¶

Official multi-architecture images are published by the release workflow to GitHub Container Registry:

• ghcr.io/lablup/continuum-router:<version> — Debian-based image
• ghcr.io/lablup/continuum-router:<version>-alpine — Alpine/musl image
• latest and latest-alpine — latest non-prerelease release

Pin a version or digest in production instead of relying on latest.

docker run --rm \
  -p 8080:8080 \
  -v "$PWD/config.yaml:/etc/continuum-router/config.yaml:ro" \
  -e OPENAI_API_KEY \
  ghcr.io/lablup/continuum-router:1.14.0

The image runs as a non-root continuum user, starts with --config /etc/continuum-router/config.yaml, and uses the built-in --health-check command for its container health check.

Docker Compose¶

The repository includes docker-compose.yml. It expects ./config.yaml and supports a VERSION environment variable:

cp config.yaml.example config.yaml
# Replace the annotated sample with your real backend/auth values, then validate it.
continuum-router config validate config.yaml
VERSION=1.14.0 docker compose up -d
docker compose logs -f continuum-router

The Compose file uses ghcr.io/lablup/continuum-router:${VERSION:-latest}. Pass provider credentials through environment variables referenced by ${ENV_VAR} placeholders in the mounted config; do not hard-code credentials into the image or Compose file.

Build repository Dockerfiles¶

Dockerfile and Dockerfile.alpine download matching GitHub release archives. Supply the release version explicitly:

docker build --build-arg VERSION=1.14.0 -t continuum-router:1.14.0 .
docker build -f Dockerfile.alpine --build-arg VERSION=1.14.0 \
  -t continuum-router:1.14.0-alpine .

They are not source-build Dockerfiles. For a customized feature set, compile the binary separately and construct an image that copies that artifact, or adapt the CI Dockerfiles with an intentional build pipeline.

Kubernetes¶

The maintained Kustomize bundle is in deploy/kubernetes/base. It includes three replicas, resource requests and limits, startup/readiness/liveness probes, a ClusterIP Service, TLS Ingress, HPA, PodDisruptionBudget, restricted pod security settings, a read-only root filesystem, and ingress/egress NetworkPolicy. The staging overlay in deploy/kubernetes/overlays/staging is used by the automated staging workflow.

Review every operator-specific value before applying it: provider/model configuration, the image tag or digest, Ingress host and TLS Secret, NetworkPolicy namespace selectors, egress rules, resource sizing, and trusted proxies. Create continuum-router-secrets separately; the checked-in bundle never creates or stores credentials.

kubectl apply -f deploy/kubernetes/base/namespace.yaml
kubectl -n continuum-router create secret generic continuum-router-secrets \
  --from-literal=OPENAI_API_KEY="$OPENAI_API_KEY" \
  --from-literal=ROUTER_API_KEY="$ROUTER_API_KEY" \
  --from-literal=ADMIN_TOKEN="$ADMIN_TOKEN"
kubectl apply -k deploy/kubernetes/base
kubectl -n continuum-router rollout status deployment/continuum-router --timeout=5m

Run scripts/validate-deployment-assets.sh after changing the manifests. It renders the base, staging overlay, monitoring bundle, and all Helm profiles; CI also validates them.

The following minimal example remains useful as an explanation of the core objects. The files under deploy/kubernetes/ are the maintained source of truth for deployment:

apiVersion: v1
kind: ConfigMap
metadata:
  name: continuum-router-config
data:
  config.yaml: |
    server:
      bind_address: 0.0.0.0:8080
      workers: 4
    selection_strategy: RoundRobin
    backends:
      - name: ollama
        type: ollama
        url: http://ollama:11434
        models: [llama3.2]
    health_checks:
      interval: 30s
      timeout: 5s
      unhealthy_threshold: 3
      healthy_threshold: 2
      endpoint: /health
    logging:
      level: info
      format: json
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: continuum-router
spec:
  replicas: 2
  selector:
    matchLabels:
      app: continuum-router
  template:
    metadata:
      labels:
        app: continuum-router
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: router
          image: ghcr.io/lablup/continuum-router:1.14.0
          args: ["--config", "/etc/continuum-router/config.yaml"]
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health
              port: http
          livenessProbe:
            httpGet:
              path: /health
              port: http
          volumeMounts:
            - name: config
              mountPath: /etc/continuum-router/config.yaml
              subPath: config.yaml
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              # Size this together with server.max_concurrent_requests, not
              # independently. See "Capacity planning: memory" below:
              # worst-case resident memory is concurrency times the
              # per-request budget. File uploads stream to disk and no longer
              # dominate that budget, but a request at the file-resolution
              # ceiling still contributes about 43 MiB once base64-expanded.
              memory: 1Gi
      volumes:
        - name: config
          configMap:
            name: continuum-router-config
---
apiVersion: v1
kind: Service
metadata:
  name: continuum-router
spec:
  selector:
    app: continuum-router
  ports:
    - name: http
      port: 8080
      targetPort: http

Use a Secret manager or Kubernetes Secret-to-environment integration for provider credentials. The ${ENV_VAR} expansion happens inside the router process.

/health is the implemented unauthenticated health endpoint. /healthz does not exist. /version is also public. Metrics use the configured metrics path and their own authentication settings.

The default readinessProbe above has no explicit periodSeconds/failureThreshold, but some orchestrators and embedding hosts poll /health with a fixed readiness deadline. By default (health_checks.block_startup: true), listener binding waits for connection pre-warm and the initial health-check round across every backend, so a single unreachable or stalling backend can push that pre-bind window past a tight deadline. Set health_checks.block_startup: false to bind listeners first and converge backend health in the background; see Startup Behavior for details.

Shutdown and rolling updates¶

The HTTP server handles SIGINT/SIGTERM and starts graceful shutdown. Set an orchestration grace period long enough for your expected streaming requests, while recognizing that the router also applies its configured shutdown timeout to background tasks and connections.

Run at least two replicas for rollout availability. Router-local state such as in-memory rate limits, response cache, stored Responses sessions, and some statistics is not automatically shared across replicas. Select Redis/persistent features where supported or accept per-instance behavior.

Helm¶

deploy/helm/continuum-router provides development, staging, canary, and production values profiles. The chart renders the same security and availability controls as the maintained Kustomize bundle and can optionally create a Prometheus Operator ServiceMonitor.

helm lint deploy/helm/continuum-router
helm template router deploy/helm/continuum-router \
  --namespace continuum-router \
  -f deploy/helm/continuum-router/values-production.yaml
kubectl create namespace continuum-router --dry-run=client -o yaml | kubectl apply -f -
kubectl label namespace continuum-router app.kubernetes.io/name=continuum-router --overwrite
helm upgrade --install router deploy/helm/continuum-router \
  --namespace continuum-router \
  -f deploy/helm/continuum-router/values-production.yaml \
  --set-string image.digest="$IMAGE_DIGEST"

Production installs should set IMAGE_DIGEST to an approved sha256:... manifest digest, replace example hosts, provide existingSecret, and review the default NetworkPolicy. Official images use per-replica in-memory rate limits. Shared Redis rate limiting and Redis response caching require a custom build with the redis-cache Cargo feature; the chart does not configure unsupported Redis behavior silently.

Automated staging deployment¶

.github/workflows/deploy-staging.yml deploys a successful published-release workflow run, or a manually selected image tag, to the continuum-router-staging namespace. Configure the protected GitHub staging environment with KUBECONFIG_B64, ROUTER_HOST, required reviewers, and deployment-branch restrictions. Create the continuum-router-staging namespace first, label it app.kubernetes.io/name=continuum-router, and then create continuum-router-secrets and continuum-router-tls there before the first run.

The workflow accepts only published semver releases, verifies the release manifest attestation, resolves the image to an immutable digest, uses a single non-cancelling concurrency group, performs a server-side apply, and waits for the Deployment rollout. All high/critical image vulnerabilities block publication, and the build publishes SBOM and provenance attestations for the final multi-architecture manifests.

Deployment strategies¶

The maintained Deployment uses a rolling update with maxUnavailable: 0; use helm rollback or reapply the previous immutable image for rollback. For blue-green deployment, install the candidate as a separate release, verify its health and metrics, then switch the ingress or upstream Service atomically while retaining the previous release during the observation window.

For a canary, install a second release with values-canary.yaml and an immutable candidate image. It creates one labeled replica without a separate Ingress, HPA, or disruption budget. Use the ingress controller or service mesh to assign a small explicit weight to that release's Service, observe error rate and latency, and promote it by upgrading the primary release. The chart does not assume a vendor-specific weighted-routing annotation. Configuration and feature rollouts can use the same two-release pattern; account for router-local sessions, caches, statistics, and rate-limit counters while releases overlap.

systemd¶

The Debian package installs the binary, examples, documentation, and manpage but no unit file. Create a local unit if systemd management is desired:

[Unit]
Description=Continuum Router
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=continuum-router
Group=continuum-router
EnvironmentFile=-/etc/continuum-router/environment
ExecStart=/usr/bin/continuum-router --config /etc/continuum-router/config.yaml
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/continuum-router /var/log/continuum-router
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Create the user and writable paths to match the features you enable, then validate before starting:

sudo /usr/bin/continuum-router config validate /etc/continuum-router/config.yaml
sudo systemctl daemon-reload
sudo systemctl enable --now continuum-router
journalctl -u continuum-router -f

For a tarball installation, adjust ExecStart to /usr/local/bin/continuum-router. Tighten or expand sandbox paths based on Files API, stats persistence, OAuth token stores, control-plane state, Unix sockets, and cache storage.

Capacity planning: memory¶

Worst-case resident memory is a product, not a sum:

worst-case memory  ~=  (concurrent requests)  x  (per-request budget)

Every memory bound the router enforces is on the right-hand factor. Nothing bounded the left one until server.max_concurrent_requests, so before you set it the ceiling is whatever your clients can open.

The per-request terms¶

Each is an independent ceiling on one request. A single request does not hit all of them at once, but each is reachable on its own.

Doing the multiplication¶

A rate limit does not supply the concurrency factor. It bounds arrivals per unit time, not the number resident at once. By Little's law:

in-flight  ~=  arrival rate  x  mean request duration

LLM completions routinely run tens of seconds, so per_client: 10 requests per second at a 30 second mean duration implies roughly 300 concurrent requests from one client, entirely within policy. At the file-resolution ceiling that is about 12.6 GiB resident; at body sizes alone, about 2.9 GiB before counting parsed JSON. Uploads no longer contribute to that figure in memory, but ten concurrent uploads at the default files.max_file_size do occupy 5 GiB of disk until they finish, so size files.storage_path against the concurrency you allow.

Setting the ceiling¶

server:
  max_concurrent_requests: 64

Unset by default, which means no ceiling and the behaviour of every earlier release. Over the ceiling the router responds 503 with a Retry-After and a JSON body naming the limit, rather than queueing: queueing would trade an out-of-memory kill for unbounded latency while still holding the memory.

Size it from the memory you are willing to spend, not from CPU:

max_concurrent_requests  ~=  (container memory limit - baseline)  /  (expected per-request cost)

Two things to know before choosing a number:

• A permit is held for the whole request, which for a streaming completion is the whole stream. A ceiling tuned against short non-streaming traffic will starve long streaming traffic. Size it against your longest requests.
• /health, /healthz, and the configured metrics.path are exempt, so a saturated router still answers its own liveness probe and stays scrapeable. Shedding the probe is how a correctly-degrading replica gets restarted in the middle of the overload it is handling.

Pair the ceiling with rate_limiting rather than choosing between them: the rate limit bounds arrival, the ceiling bounds residency, and neither substitutes for the other. A reverse proxy in front helps only if it is explicitly configured to cap concurrent connections and body size, and it cannot know the per-request cost of a request carrying 100 file references.

Pair it with a request timeout as well. The router applies timeouts to its upstream calls, not to how long an inbound client may hold a slot. A client that opens a streaming completion and then stops reading pins its permit for as long as the connection lives, so a ceiling on its own converts a memory-exhaustion attack (expensive for the attacker) into a slot-exhaustion one (cheap: N idle sockets). Enforce an inbound request or idle timeout at your proxy or ingress.

One reporting caveat if you consume the control-plane supply feed: max_concurrency is the ceiling across every route, while active_requests counts only the API routes. A router saturated by file uploads therefore shows a low active_requests against a full ceiling, so do not read the ratio as utilization.

Sizing the container¶

The Kubernetes example above requests 256Mi and limits 1Gi. Treat those numbers as a floor for an idle router, and raise both together with the ceiling you set. One request at the file-resolution ceiling still contributes about 43 MiB once base64-expanded, so the concurrency ceiling and the container memory limit are one decision. For example, max_concurrent_requests: 64 with an expected 10 MiB per request wants roughly 1Gi of headroom above baseline.

files.max_file_size is a disk budget rather than a memory one: a deployment that accepts uploads should size files.storage_path against that value times the concurrency it allows, and the router warns at startup if files.max_file_size is large relative to the container memory limit, because a few consumers (base64 file injection into chat requests, provider passthrough re-upload) still handle a whole file at once. The chat-completions base64 injection path checks stored metadata before reading content and refuses files above 10 MiB or requests above 32 MiB raw, so larger uploads may still exist on disk but cannot be expanded into one chat request.

Under an orchestrator with a memory limit set, exceeding it is an OOM kill and a restart: the blast radius is that replica's availability plus every in-flight request on it. On bare metal, or in a container with no memory limit, it is host-level pressure instead.

High availability¶

Continuum Router has no regions:, geographic_routing:, failover:, or postgresql: top-level configuration. Implement cross-region traffic management and TLS with external infrastructure.

Within one router instance, health filtering, circuit breaking, retry, model fallback, and the six selection strategies provide backend-level resilience. Ordinary proxy traffic records outcomes and routes around open backend circuits. Across router replicas:

• Use an external load balancer with /health checks.
• Keep configuration and secret versions consistent.
• Understand which rate-limit/cache/statistics stores are per process.
• Avoid assuming stored /v1/responses/{id} sessions can be retrieved from another replica without sticky ingress; current session storage is process-local.
• Plan Files API storage and ownership metadata for shared access before scaling it horizontally.

Performance profile¶

A safe tuning workflow is:

1. Start from a generated template.
2. Measure direct-provider and routed traffic.
3. Tune server.connection_pool_size, timeouts, retries, health checks, and selection_strategy.
4. Size response cache and rate limits from observed traffic.
5. Validate every change. Selection strategy, retry policy, and per-request timeout budgets reload live; restart for startup-built fields such as listener/client construction (timeouts.connection and the shared-client ceiling at timeouts.request.streaming.total).

Security¶

• Use API-key blocking mode for public API traffic.
• Configure Admin, Files, WebUI, and Metrics authentication independently.
• Terminate TLS before the router.
• Keep provider keys in environment/secret stores and reference them with ${ENV_VAR}.
• Bind Admin access to a private network or strict proxy policy.
• Do not publish config show --resolved output; it may contain plaintext credentials.
• Mount configuration read-only unless using Admin configuration save/import operations, which require a writable destination.

The public /health and /version endpoints contain operational information and do not use API-key authentication. Apply network-layer filtering if even that information must be private.

Observability¶

Enable metrics explicitly and protect them:

metrics:
  enabled: true
  endpoint: /metrics
  auth:
    enabled: true
    username: metrics
    password: "${METRICS_PASSWORD}"

Use logging.format: json for structured production logs. CONTINUUM_LOG_LEVEL and RUST_LOG can adjust verbosity; avoid debug logging of sensitive workloads for long periods.

Admin diagnostics include:

curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:8080/admin/health
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:8080/admin/backends
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:8080/admin/circuit/all

Backup and recovery¶

Back up the operator-owned state that your configuration actually enables:

• config.yaml or TOML source, kept in version control without secrets
• External API-key file, if configured
• Files API storage and metadata database
• Stats/metrics persistence files
• OAuth token stores
• Control-plane state and its probe-budget sidecar when the feature is active
• Response-cache backing stores when persistence matters

Test restore procedures with the same binary version and feature set. Validate restored configuration before starting.

Troubleshooting¶

Container is unhealthy¶

docker exec continuum-router continuum-router --health-check \
  --health-check-url http://localhost:8080/health
docker logs continuum-router

Confirm the router is listening on 0.0.0.0:8080 inside a container rather than loopback only.

Configuration fails¶

continuum-router config validate /etc/continuum-router/config.yaml
continuum-router config show /etc/continuum-router/config.yaml

Use config show --resolved only in a protected terminal because it can reveal resolved secrets.

No backend is selected¶

Check the requested model, backend model lists, health, circuit state, and caller allow-lists. Selection strategies only choose among eligible candidates.

See also¶

• Installation
• Configuration
• Performance
• Security
• Metrics