Deployment Guide¶

This guide covers deployment surfaces that exist in the repository: release binaries, official container images, Docker Compose, maintained Kubernetes/Kustomize assets, the Helm chart, and custom systemd units. The Debian package does not install a systemd unit.

Before deployment¶

Create and validate a configuration:

continuum-router config generate --template production-ha --output config.yaml
# Set required environment variables referenced by the template.
continuum-router config validate config.yaml

For a smaller starting point, use --template minimal. Available templates are minimal, development, multi-provider, production-ha, api-gateway, kv-cache-optimized, smart-routing, disaggregated, cost-optimized, sglang, and ab-testing.

Bind to loopback unless an external listener is required. Enable API and Admin authentication before exposing the service to an untrusted network. The router serves plain HTTP; terminate TLS in a trusted reverse proxy, ingress controller, or load balancer.

Release feature set¶

A default source build uses Cargo's full feature set. Official release binaries and official container images additionally compile control-plane, appproxy-router, and redis-cache; all three remain runtime opt-in. appproxy-legacy, bedrock-sigv4, and s3-cache remain source-build opt-ins unless a release workflow explicitly adds them.

redis-cache joined the official set so a multi-replica deployment can share rate-limit counters and the response cache without a custom build. Compiling it activates nothing: the router contacts Redis only when rate_limiting.storage: redis or response_cache.backend: redis names an endpoint. See Shared Redis state.

Runtime configuration cannot enable code that was not compiled. Check /admin/capabilities with Admin credentials when diagnosing a feature-gated deployment.

Docker¶

One multi-architecture image is published by the release workflow to GitHub Container Registry:

• ghcr.io/lablup/continuum-router:<version>: pinned version
• ghcr.io/lablup/continuum-router:<major>.<minor>: latest patch of that minor
• latest: latest non-prerelease release

Pin a version or digest in production instead of relying on latest.

The image is built on gcr.io/distroless/static-debian12 and carries the statically linked musl binary, a passwd file, and timezone data. There is no shell and no package manager. The router links rustls with bundled trust roots, so outbound HTTPS to providers works without a system certificate store, and no CA certificate package is installed. See Getting a shell for debugging a container that has no shell of its own.

docker run --rm \
  -p 8080:8080 \
  -v "$PWD/config.yaml:/etc/continuum-router/config.yaml:ro" \
  -e OPENAI_API_KEY \
  ghcr.io/lablup/continuum-router:1.28.0

The image runs as uid 65532 (the nonroot user distroless defines), starts with --config /etc/continuum-router/config.yaml, and uses the built-in --health-check command for its container health check.

Docker Compose¶

The repository includes docker-compose.yml. It expects ./config.yaml and supports a VERSION environment variable:

cp config.yaml.example config.yaml
# Replace the annotated sample with your real backend/auth values, then validate it.
continuum-router config validate config.yaml
VERSION=1.28.0 docker compose up -d
docker compose logs -f continuum-router

The Compose file uses ghcr.io/lablup/continuum-router:${VERSION:-latest}. Pass provider credentials through environment variables referenced by ${ENV_VAR} placeholders in the mounted config; do not hard-code credentials into the image or Compose file.

Build repository Dockerfiles¶

Dockerfile downloads the matching GitHub release archive. Supply the release version explicitly:

docker build --build-arg VERSION=1.28.0 -t continuum-router:1.28.0 .

It is not a source-build Dockerfile. For a customized feature set, compile the binary separately and construct an image that copies that artifact, or adapt Dockerfile.ci with an intentional build pipeline.

Kubernetes¶

The maintained Kustomize bundle is in deploy/kubernetes/base. It includes three replicas, resource requests and limits, startup/readiness/liveness probes, a ClusterIP Service, TLS Ingress, HPA, PodDisruptionBudget, restricted pod security settings, a read-only root filesystem, and ingress/egress NetworkPolicy. The staging overlay in deploy/kubernetes/overlays/staging is used by the automated staging workflow.

Review every operator-specific value before applying it: provider/model configuration, the image tag or digest, Ingress host and TLS Secret, NetworkPolicy namespace selectors, egress rules, resource sizing, and trusted proxies. Create continuum-router-secrets separately; the checked-in bundle never creates or stores credentials.

kubectl apply -f deploy/kubernetes/base/namespace.yaml
kubectl -n continuum-router create secret generic continuum-router-secrets \
  --from-literal=OPENAI_API_KEY="$OPENAI_API_KEY" \
  --from-literal=ROUTER_API_KEY="$ROUTER_API_KEY" \
  --from-literal=ADMIN_TOKEN="$ADMIN_TOKEN"
kubectl apply -k deploy/kubernetes/base
kubectl -n continuum-router rollout status deployment/continuum-router --timeout=5m

Run scripts/validate-deployment-assets.sh after changing the manifests. It renders the base, staging overlay, monitoring bundle, and all Helm profiles; CI also validates them.

The following minimal example remains useful as an explanation of the core objects. The files under deploy/kubernetes/ are the maintained source of truth for deployment:

apiVersion: v1
kind: ConfigMap
metadata:
  name: continuum-router-config
data:
  config.yaml: |
    server:
      bind_address: 0.0.0.0:8080
      workers: 4
    selection_strategy: RoundRobin
    backends:
      - name: ollama
        type: ollama
        url: http://ollama:11434
        models: [llama3.2]
    health_checks:
      interval: 30s
      timeout: 5s
      unhealthy_threshold: 3
      healthy_threshold: 2
      endpoint: /health
    logging:
      level: info
      format: json
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: continuum-router
spec:
  replicas: 2
  selector:
    matchLabels:
      app: continuum-router
  template:
    metadata:
      labels:
        app: continuum-router
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: router
          image: ghcr.io/lablup/continuum-router:1.28.0
          args: ["--config", "/etc/continuum-router/config.yaml"]
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health
              port: http
          livenessProbe:
            httpGet:
              path: /health
              port: http
          volumeMounts:
            - name: config
              mountPath: /etc/continuum-router/config.yaml
              subPath: config.yaml
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              # Size this together with server.max_concurrent_requests, not
              # independently. See "Capacity planning: memory" below:
              # worst-case resident memory is concurrency times the
              # per-request budget. File uploads stream to disk and no longer
              # dominate that budget, but a request at the file-resolution
              # ceiling still contributes about 43 MiB once base64-expanded.
              memory: 1Gi
      volumes:
        - name: config
          configMap:
            name: continuum-router-config
---
apiVersion: v1
kind: Service
metadata:
  name: continuum-router
spec:
  selector:
    app: continuum-router
  ports:
    - name: http
      port: 8080
      targetPort: http

Use a Secret manager or Kubernetes Secret-to-environment integration for provider credentials. The ${ENV_VAR} expansion happens inside the router process.

/health is the implemented unauthenticated health endpoint. /healthz does not exist. /version is also public. Metrics use the configured metrics path and their own authentication settings.

The default readinessProbe above has no explicit periodSeconds/failureThreshold, but some orchestrators and embedding hosts poll /health with a fixed readiness deadline. By default (health_checks.block_startup: true), listener binding waits for connection pre-warm and the initial health-check round across every backend, so a single unreachable or stalling backend can push that pre-bind window past a tight deadline. Set health_checks.block_startup: false to bind listeners first and converge backend health in the background; see Startup Behavior for details.

Shutdown and rolling updates¶

The HTTP server handles SIGINT/SIGTERM and starts graceful shutdown. Set an orchestration grace period long enough for your expected streaming requests, while recognizing that the router also applies its configured shutdown timeout to background tasks and connections.

Run at least two replicas for rollout availability. Router-local state such as in-memory rate limits, response cache, stored Responses sessions, and some statistics is not automatically shared across replicas. Select Redis/persistent features where supported or accept per-instance behavior.

Helm¶

deploy/helm/continuum-router provides development, staging, canary, and production values profiles. The chart renders the same security and availability controls as the maintained Kustomize bundle and can optionally create a Prometheus Operator ServiceMonitor.

helm lint deploy/helm/continuum-router
helm template router deploy/helm/continuum-router \
  --namespace continuum-router \
  -f deploy/helm/continuum-router/values-production.yaml
kubectl create namespace continuum-router --dry-run=client -o yaml | kubectl apply -f -
kubectl label namespace continuum-router app.kubernetes.io/name=continuum-router --overwrite
helm upgrade --install router deploy/helm/continuum-router \
  --namespace continuum-router \
  -f deploy/helm/continuum-router/values-production.yaml \
  --set-string image.digest="$IMAGE_DIGEST"

Production installs should set IMAGE_DIGEST to an approved sha256:... manifest digest, replace example hosts, provide existingSecret, and review the default NetworkPolicy. The chart defaults to per-replica in-memory rate limits. Layer values-shared-redis.yaml on top of a profile to switch to shared state; see Shared Redis state.

Automated staging deployment¶

.github/workflows/deploy-staging.yml deploys a successful published-release workflow run, or a manually selected image tag, to the continuum-router-staging namespace. Configure the protected GitHub staging environment with KUBECONFIG_B64, ROUTER_HOST, required reviewers, and deployment-branch restrictions. Create the continuum-router-staging namespace first, label it app.kubernetes.io/name=continuum-router, and then create continuum-router-secrets and continuum-router-tls there before the first run.

The workflow accepts only published semver releases, verifies the release manifest attestation, resolves the image to an immutable digest, uses a single non-cancelling concurrency group, performs a server-side apply, and waits for the Deployment rollout. All high/critical image vulnerabilities block publication, and the build publishes SBOM and provenance attestations for the final multi-architecture manifests.

Deployment strategies¶

The maintained Deployment uses a rolling update with maxUnavailable: 0; use helm rollback or reapply the previous immutable image for rollback. For blue-green deployment, install the candidate as a separate release, verify its health and metrics, then switch the ingress or upstream Service atomically while retaining the previous release during the observation window.

For a canary, install a second release with values-canary.yaml and an immutable candidate image. It creates one labeled replica without a separate Ingress, HPA, or disruption budget. Use the ingress controller or service mesh to assign a small explicit weight to that release's Service, observe error rate and latency, and promote it by upgrading the primary release. The chart does not assume a vendor-specific weighted-routing annotation. Configuration and feature rollouts can use the same two-release pattern; account for router-local sessions, caches, statistics, and rate-limit counters while releases overlap.

Automated progressive delivery with Argo Rollouts¶

The strategies above are controller-independent and stay that way: the base chart, the Kustomize bundle, values-canary.yaml, and the blue-green two-release switch all work with nothing installed beyond Kubernetes. What they do not give you is automation. Traffic shifting, metric analysis, promotion, and rollback are all manual, which is fine for a planned release and poor at 3 a.m.

The chart can hand that work to Argo Rollouts. It is off by default and gated on one value.

Enabling it¶

The controller and its CRDs must exist in the cluster first. Without them the API server rejects the Rollout object:

kubectl get crd rollouts.argoproj.io analysistemplates.argoproj.io

Then layer the overlay on an environment profile:

helm upgrade --install router deploy/helm/continuum-router \
  --namespace continuum-router \
  -f deploy/helm/continuum-router/values-production.yaml \
  -f deploy/helm/continuum-router/values-progressive-delivery.yaml \
  --set-string image.digest="$IMAGE_DIGEST"

What changes in the rendered manifests:

The two workload objects share one pod template (continuum-router.podTemplate in _helpers.tpl), so a security context, probe, or volume added to one is added to the other. They cannot drift.

The HPA retarget matters: an HPA still pointing at a Deployment that no longer exists sits inert and the router never scales. Pointed at the Rollout, Argo Rollouts consumes the desired replica count and distributes it across the stable and canary ReplicaSets itself.

Weighted canary steps¶

progressiveDelivery.argoRollouts.steps is passed to strategy.canary.steps verbatim, so any step Argo Rollouts understands is available. The production overlay ships 10%, 30%, 60% with a ten-minute soak between each.

Without a traffic-routing provider the weight is approximated by ReplicaSet pod counts behind the single Service. That is accurate enough to observe a canary and is the most the generic chart can guarantee, since it assumes no particular mesh or ingress. For enforced weighting, set progressiveDelivery.argoRollouts.trafficRouting.enabled: true and put the provider block in trafficRouting.spec; the chart then renders <release>-canary and <release>-stable Services and passes the block through:

progressiveDelivery:
  argoRollouts:
    trafficRouting:
      enabled: true
      spec:
        nginx:
          stableIngress: continuum-router

Analysis¶

AnalysisTemplate <release>-health runs two measurements against the router's own metrics, scoped to the canary ReplicaSet:

• error-rate: sum(rate(errors_total[2m])) / sum(rate(http_requests_total[2m])), must stay at or below analysis.maxErrorRate (0.05).
• latency-p95: histogram_quantile(0.95, ...http_request_duration_seconds_bucket...), must stay at or below analysis.maxLatencyP95Seconds (2.0).

Scoping to the canary is what makes the measurement meaningful, and it depends on one label. The chart's ServiceMonitor relabels rollouts-pod-template-hash from the pod onto every series as rollouts_pod_template_hash, and the standalone bundle in monitoring/prometheus/ carries the same rule. A Prometheus configured elsewhere needs it too, or the analysis measures the fleet average and a bad canary hides inside good stable traffic:

relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_rollouts_pod_template_hash]
    target_label: rollouts_pod_template_hash

Both queries end with and sum(rate(http_requests_total{...})) > analysis.minRequestRate. Below that rate the query returns an empty vector, neither the success nor the failure condition matches, and Argo Rollouts records the measurement as Inconclusive. An inconclusive run pauses for an operator rather than deciding. This is the low-traffic guard, and it is the difference between a canary judged on evidence and one judged on three overnight requests: a single failure out of three is a 33% error rate, which would roll back a perfectly good release. analysis.inconclusiveLimit (3) bounds how long it waits before pausing.

Analysis runs in the background from analysis.startingStep onward, not only between steps, so a canary that degrades mid-soak aborts immediately instead of at the next step boundary. The default 1 is a zero-based index into steps, which puts the first measurement in the first soak, after the first weight has been applied and traffic is actually reaching the canary. Starting at 0 would measure before any weight is set, where there is nothing to measure.

Tuning, in the order worth trying:

Promotion¶

With manualPromotionPause: true (the production overlay) the rollout stops after the last weighted step and waits indefinitely. Promotion is a deliberate act:

kubectl argo rollouts get rollout router-continuum-router --watch
kubectl argo rollouts promote router-continuum-router

The pause blocks promotion only. Analysis keeps running through it, so a canary that degrades while waiting for approval still aborts on its own. Set manualPromotionPause: false where an unattended promotion is acceptable, for example in staging.

Rollback¶

Failed analysis aborts the rollout automatically: the canary ReplicaSet scales to zero and the stable ReplicaSet keeps serving, which means the previously running version was never taken out of service. Manually:

kubectl argo rollouts abort router-continuum-router     # stop and return traffic to stable
kubectl argo rollouts undo router-continuum-router      # roll back to the previous revision

Drill the failure path before you depend on it. On a staging install, deploy an image that fails readiness or returns errors, watch the canary weight stop advancing, and confirm with kubectl argo rollouts get rollout <name> that the analysis run is Failed, the rollout is Degraded, and the stable ReplicaSet still has its full replica count. That last check is the one that matters: an abort that also scaled down stable would be an outage, not a rollback.

Configuration rollouts¶

A configuration change is delivered by the same mechanism, and it needs no extra setting. The pod template carries a checksum/config annotation over the rendered ConfigMap, so editing routerConfig (or the ConfigMap named by existingConfigMap) changes the pod template hash, which starts a fresh canary progression through the same steps and the same analysis. A bad limit, a bad timeout, or a bad backend inventory is caught on 10% of traffic and rolled back automatically, exactly like a bad image.

Two things this does not cover. Settings the router hot-reloads in place are applied by the running process without a new pod, so they bypass the rollout entirely; deliver those through a config revision (a ConfigMap change) when you want them gated. And Cargo features are compile-time, so "feature rollout" always means rolling out a configuration revision or an image, never flipping a runtime flag; see Release feature set.

Recovering to a plain Deployment¶

Progressive delivery is reversible. Reinstall without the overlay:

helm upgrade --install router deploy/helm/continuum-router \
  --namespace continuum-router \
  -f deploy/helm/continuum-router/values-production.yaml \
  --set-string image.digest="$IMAGE_DIGEST"
kubectl -n continuum-router delete rollout router-continuum-router
kubectl -n continuum-router rollout status deployment/router-continuum-router --timeout=5m

Delete the Rollout after the Deployment is up, not before: the order keeps pods serving throughout. If the controller itself is broken or being uninstalled, this is also the escape hatch, and it needs nothing from the controller to work.

Flagger¶

The chart does not ship Flagger resources, but the mapping is direct if that is your ecosystem: a Canary with analysis.steps replaces the Rollout, analysis.metrics with a MetricTemplate replaces the AnalysisTemplate, and the same two PromQL queries and the same low-traffic guard apply unchanged. Flagger keeps the Deployment and generates its own ReplicaSets, so the deployment guard here is not needed; leave progressiveDelivery.argoRollouts.enabled: false and install the Canary alongside the chart.

systemd¶

The Debian package installs the binary, examples, documentation, and manpage but no unit file. Create a local unit if systemd management is desired:

[Unit]
Description=Continuum Router
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=continuum-router
Group=continuum-router
EnvironmentFile=-/etc/continuum-router/environment
ExecStart=/usr/bin/continuum-router --config /etc/continuum-router/config.yaml
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/continuum-router /var/log/continuum-router
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Create the user and writable paths to match the features you enable, then validate before starting:

sudo /usr/bin/continuum-router config validate /etc/continuum-router/config.yaml
sudo systemctl daemon-reload
sudo systemctl enable --now continuum-router
journalctl -u continuum-router -f

For a tarball installation, adjust ExecStart to /usr/local/bin/continuum-router. Tighten or expand sandbox paths based on Files API, stats persistence, OAuth token stores, control-plane state, Unix sockets, and cache storage.

Capacity planning: memory¶

Worst-case resident memory is a product, not a sum:

worst-case memory  ~=  (concurrent requests)  x  (per-request budget)

Every memory bound the router enforces is on the right-hand factor. Nothing bounded the left one until server.max_concurrent_requests, so before you set it the ceiling is whatever your clients can open.

The per-request terms¶

Each is an independent ceiling on one request. A single request does not hit all of them at once, but each is reachable on its own.

Doing the multiplication¶

A rate limit does not supply the concurrency factor. It bounds arrivals per unit time, not the number resident at once. By Little's law:

in-flight  ~=  arrival rate  x  mean request duration

LLM completions routinely run tens of seconds, so per_client: 10 requests per second at a 30 second mean duration implies roughly 300 concurrent requests from one client, entirely within policy. At the file-resolution ceiling that is about 12.6 GiB resident; at body sizes alone, about 2.9 GiB before counting parsed JSON. Uploads no longer contribute to that figure in memory, but ten concurrent uploads at the default files.max_file_size do occupy 5 GiB of disk until they finish, so size files.storage_path against the concurrency you allow.

Setting the ceiling¶

server:
  max_concurrent_requests: 64

Unset by default, which means no ceiling and the behaviour of every earlier release. Over the ceiling the router responds 503 with a Retry-After and a JSON body naming the limit, rather than queueing: queueing would trade an out-of-memory kill for unbounded latency while still holding the memory.

Size it from the memory you are willing to spend, not from CPU:

max_concurrent_requests  ~=  (container memory limit - baseline)  /  (expected per-request cost)

Two things to know before choosing a number:

• A permit is held for the whole request, which for a streaming completion is the whole stream. A ceiling tuned against short non-streaming traffic will starve long streaming traffic. Size it against your longest requests.
• /health, /healthz, and the configured metrics.path are exempt, so a saturated router still answers its own liveness probe and stays scrapeable. Shedding the probe is how a correctly-degrading replica gets restarted in the middle of the overload it is handling.

Pair the ceiling with rate_limiting rather than choosing between them: the rate limit bounds arrival, the ceiling bounds residency, and neither substitutes for the other. A reverse proxy in front helps only if it is explicitly configured to cap concurrent connections and body size, and it cannot know the per-request cost of a request carrying 100 file references.

Pair it with a request timeout as well. The router applies timeouts to its upstream calls, not to how long an inbound client may hold a slot. A client that opens a streaming completion and then stops reading pins its permit for as long as the connection lives, so a ceiling on its own converts a memory-exhaustion attack (expensive for the attacker) into a slot-exhaustion one (cheap: N idle sockets). Enforce an inbound request or idle timeout at your proxy or ingress.

One reporting caveat if you consume the control-plane supply feed: max_concurrency is the ceiling across every route, while active_requests counts only the API routes. A router saturated by file uploads therefore shows a low active_requests against a full ceiling, so do not read the ratio as utilization.

Sizing the container¶

The Kubernetes example above requests 256Mi and limits 1Gi. Treat those numbers as a floor for an idle router, and raise both together with the ceiling you set. One request at the file-resolution ceiling still contributes about 43 MiB once base64-expanded, so the concurrency ceiling and the container memory limit are one decision. For example, max_concurrent_requests: 64 with an expected 10 MiB per request wants roughly 1Gi of headroom above baseline.

files.max_file_size is a disk budget rather than a memory one: a deployment that accepts uploads should size files.storage_path against that value times the concurrency it allows, and the router warns at startup if files.max_file_size is large relative to the container memory limit, because a few consumers (base64 file injection into chat requests, provider passthrough re-upload) still handle a whole file at once. The chat-completions base64 injection path checks stored metadata before reading content and refuses files above 10 MiB or requests above 32 MiB raw, so larger uploads may still exist on disk but cannot be expanded into one chat request.

Under an orchestrator with a memory limit set, exceeding it is an OOM kill and a restart: the blast radius is that replica's availability plus every in-flight request on it. On bare metal, or in a container with no memory limit, it is host-level pressure instead.

Capacity planning: throughput¶

The section above bounds what one replica may hold. This one is about what it can serve, and about the shipped defaults that depend on that: the 200m CPU request and 1 core limit in deploy/kubernetes/base/deployment.yaml, and the 70% CPU target in the HorizontalPodAutoscaler.

Measure it, do not read it off a page¶

Router throughput is not a constant. It is a function of your upstream provider latency, your streaming mix, the request shape, and the CPU you give a replica, and only the last of those is under this project's control. A headline requests-per-second figure published here would be wrong for almost every deployment. So the repository ships the measurement instead:

perf/run.sh smoke      # three seconds, confirms the harness works
perf/run.sh            # both capacity profiles, three trials each, then the gate

Nothing external is involved. The harness starts its own mock backends with a configured first-token delay and inter-token cadence, generates a router configuration pointing at them, starts the router as a separate process, drives sustained load over real sockets, and writes JSON containing throughput, latency percentiles, time to first token, error rate, router process RSS and CPU, and backend-pool occupancy.

Two profiles ship: non-streaming (32 connections, upstream fixed at 60ms) and streaming (32 connections, 64 tokens at 5ms). perf/README.md documents the profiles and the result schema. .github/workflows/perf.yml runs the same profiles weekly against reviewed thresholds in perf/thresholds.json, on a pinned runner class, and never on a per-commit build.

Run it on hardware matching your target replica, with the request shape you actually serve. A measurement on a workstation tells you nothing about a 200m CPU request.

Turning a measurement into a replica count¶

Two facts do most of the sizing work.

Concurrency, not request rate, sets the replica count. By Little's law, in-flight ~= arrival rate x mean duration. A completion that runs 20 seconds at 5 requests per second is 100 concurrent requests, and it is those 100 slots a replica has to hold, not the 5 per second. Read throughput_rps from the non-streaming profile as the ceiling on the router's own request-path work at the offered concurrency, and treat load above it as needing another replica rather than a bigger one.

The router relays streams rather than buffering them. The streaming profile measures this directly, and it is the reason a streaming replica is sized differently: with a backend that emits its first token at 50ms and finishes the stream about 315ms later, the measured time to first token stays near the upstream first-token delay rather than near the full stream duration. So the cost of a streaming replica is dominated by how many streams are open at once, and each open stream holds its max_concurrent_requests permit for the stream's entire life. Size against concurrent streams and their duration, not against completed requests per second.

The two calculations are separate and both are needed:

• Throughput tells you how many replicas. Divide your expected peak concurrency by the concurrency at which you measured.
• Memory tells you how large each replica is. Use the previous section, not this one, to choose server.max_concurrent_requests and the container memory limit.

Validating the autoscaling defaults on a cluster¶

The bundle ships minReplicas: 3 and maxReplicas: 10 (20 in the Helm production values) against a 70% CPU utilization target. Those defaults assume a replica becomes CPU-bound before it becomes memory-bound. Whether that holds is a property of your workload, and it is worth confirming before you rely on it:

1. Establish a per-replica baseline inside the cluster. Run the single-replica profile from a Job on the same node class as the router, so the measurement inherits the cluster's CPU limits rather than a laptop's.
2. Scale the offered load. Point the driver at the Service with continuum-perf run --target-url <service-host>:<port>, and raise offered concurrency past one replica's measured ceiling. In --target-url mode the harness cannot read /proc for an in-cluster process, so take memory and CPU from kubectl top pod instead.
3. Watch the autoscaler and the latency together. kubectl get hpa continuum-router --watch alongside the driver's p95. The defaults hold if replicas rise before p95 leaves the single-replica band, if the 30 second scale-up stabilization window is short enough for latency to recover inside it, and if the 300 second scale-down window does not thrash under sawtooth load.
4. Check the failure mode that matters. If CPU utilization stays well under 70% while latency climbs, the replica is not CPU-bound at your workload shape, and a CPU-target HPA will not scale it at all. That is the common case for a streaming-heavy deployment, where a replica is bound by open streams rather than by compute. Lower averageUtilization, or scale on a custom metric such as in-flight requests, rather than raising maxReplicas and hoping.

Record what you measure. A capacity number without the request shape, the upstream latency, and the replica size that produced it is not reproducible, which is exactly why the harness writes all three into every result file.

High availability¶

Continuum Router has no regions:, geographic_routing:, failover:, or postgresql: top-level configuration. Implement cross-region traffic management and TLS with external infrastructure.

Within one router instance, health filtering, circuit breaking, retry, model fallback, and the six selection strategies provide backend-level resilience. Ordinary proxy traffic records outcomes and routes around open backend circuits. Across router replicas:

• Use an external load balancer with /health checks.
• Keep configuration and secret versions consistent.
• Understand which rate-limit/cache/statistics stores are per process.
• Avoid assuming stored /v1/responses/{id} sessions can be retrieved from another replica without sticky ingress; current session storage is process-local.
• Plan Files API storage and ownership metadata for shared access before scaling it horizontally.

Shared Redis state¶

Rate-limit counters and the response cache are per process by default, so a three-replica deployment admits three times the configured rate and caches the same response three times. Pointing every replica at one Redis or Valkey endpoint removes both. Official binaries and images compile redis-cache, so this needs configuration only.

Supported topology¶

The boundary follows from the client: the router builds one deadpool-redis pool over redis::aio::ConnectionManager, which discovers no Sentinel master and routes no cluster slots. A Sentinel or Cluster URL would either fail to connect or, worse, pin every replica to whichever single node answered. Rather than imply support, the configuration loader refuses those URLs with a message naming the limitation, at startup, at hot reload, through the admin config API, and in continuum-router config validate.

Configuration¶

rate_limiting:
  enabled: true
  storage: redis
  redis:
    url: "${REDIS_URL}"
    key_prefix: "continuum:ratelimit:"
    ttl: 3600

response_cache:
  enabled: true
  backend: redis
  ttl: "300s"
  redis:
    url: "${REDIS_URL}"
    pool_size: 8
    key_prefix: "cr:resp:"
    fallback_to_memory: true

TLS comes from the rediss:// scheme (the response cache also accepts tls: true, which rewrites the scheme). Credentials belong in the URL userinfo and should arrive through the environment, never in a checked-in file. config validate warns about a password written into the config and warns again when credentials travel over a plaintext redis:// endpoint; router logs always mask the password.

Deployment wiring¶

Neither the Helm chart nor the Kustomize bundle ships a Redis server, a StatefulSet, or a generated credential. They reference a Secret you create and open the NetworkPolicy egress needed to reach the endpoint.

kubectl -n continuum-router create secret generic continuum-router-redis \
  --from-literal=REDIS_URL="rediss://:$REDIS_PASSWORD@cache.example.com:6380"

# Helm: layer the overlay on an environment profile.
helm upgrade --install router deploy/helm/continuum-router \
  --namespace continuum-router \
  -f deploy/helm/continuum-router/values-production.yaml \
  -f deploy/helm/continuum-router/values-shared-redis.yaml \
  --set-string image.digest="$IMAGE_DIGEST"

# Kustomize: apply the overlay instead of the base.
kubectl apply -k deploy/kubernetes/overlays/shared-redis

Both overlays open egress to port 6380 by port number only, because a managed endpoint may live outside the cluster and cannot be selected by namespace or pod label. Narrow that to an ipBlock once you know the address. redis.enabled: true without a Secret name fails the Helm render rather than inventing a credential.

Failure semantics¶

Neither consumer refuses traffic when Redis disappears. Size rate limits against the degraded ceiling: if you run six replicas and the fleet must never exceed 600 rps, an outage with a 600 rps shared limit admits 3600 rps. Reconnection is automatic in both consumers, and no restart is needed after the endpoint returns.

Recovering from a shared-state deployment back to per-replica state is a config change: set storage: memory and backend: memory, reapply, and roll. Counters restart from empty.

Performance profile¶

A safe tuning workflow is:

1. Start from a generated template.
2. Measure direct-provider and routed traffic. For the routed side, perf/run.sh gives a reproducible measurement without a provider account (see "Capacity planning: throughput" above).
3. Tune server.connection_pool_size, timeouts, retries, health checks, and selection_strategy.
4. Size response cache and rate limits from observed traffic.
5. Validate every change. Selection strategy, retry policy, and per-request timeout budgets reload live; restart for startup-built fields such as listener/client construction (timeouts.connection and the shared-client ceiling at timeouts.request.streaming.total).

Security¶

• Use API-key blocking mode for public API traffic.
• Configure Admin, Files, WebUI, and Metrics authentication independently.
• Terminate TLS before the router.
• Keep provider keys in environment/secret stores and reference them with ${ENV_VAR}.
• Bind Admin access to a private network or strict proxy policy.
• Do not publish config show --resolved output; it may contain plaintext credentials.
• Mount configuration read-only unless using Admin configuration save/import operations, which require a writable destination.

The public /health and /version endpoints contain operational information and do not use API-key authentication. Apply network-layer filtering if even that information must be private.

Observability¶

Enable metrics explicitly and protect them:

metrics:
  enabled: true
  endpoint: /metrics
  auth:
    enabled: true
    username: metrics
    password: "${METRICS_PASSWORD}"

Use logging.format: json for structured production logs. CONTINUUM_LOG_LEVEL and RUST_LOG can adjust verbosity; avoid debug logging of sensitive workloads for long periods.

Admin diagnostics include:

curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:8080/admin/health
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:8080/admin/backends
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:8080/admin/circuit/all

Backup and recovery¶

Back up the operator-owned state that your configuration actually enables:

• config.yaml or TOML source, kept in version control without secrets
• External API-key file, if configured
• Files API storage and metadata database
• Stats/metrics persistence files
• OAuth token stores
• Control-plane state and its probe-budget sidecar when the feature is active
• Response-cache backing stores when persistence matters

Test restore procedures with the same binary version and feature set. Validate restored configuration before starting.

Troubleshooting¶

Getting a shell¶

The image ships no shell, so docker exec ... sh and kubectl exec ... -- sh do not work. Attach a shell in a separate container that shares the router's namespaces instead. The router's own filesystem is reachable at /proc/1/root.

# Kubernetes: an ephemeral debug container in the running pod
kubectl debug -it <pod> --image=busybox --target=continuum-router

# Docker: a throwaway container sharing the router's PID and network namespaces
docker run --rm -it \
  --pid=container:continuum-router \
  --network=container:continuum-router \
  busybox sh

Most inspection needs no shell at all, because the router's own subcommands run as the image entrypoint:

docker run --rm -v "$PWD/config.yaml:/c.yaml:ro" \
  ghcr.io/lablup/continuum-router:1.28.0 config validate /c.yaml
docker run --rm -v "$PWD/config.yaml:/c.yaml:ro" \
  ghcr.io/lablup/continuum-router:1.28.0 config show --resolved /c.yaml

Container is unhealthy¶

docker exec continuum-router /usr/local/bin/continuum-router --health-check
docker logs continuum-router

Confirm the router is listening on 0.0.0.0:8080 inside a container rather than loopback only.

Configuration fails¶

continuum-router config validate /etc/continuum-router/config.yaml
continuum-router config show /etc/continuum-router/config.yaml

Use config show --resolved only in a protected terminal because it can reveal resolved secrets.

No backend is selected¶

Check the requested model, backend model lists, health, circuit state, and caller allow-lists. Selection strategies only choose among eligible candidates.

See also¶

• Installation
• Configuration
• Performance
• Security
• Metrics