Deployment Guide¶

This guide covers deployment surfaces that exist in the repository: release binaries, official container images, Docker Compose, maintained Kubernetes/Kustomize assets, the Helm chart, and custom systemd units. The Debian package does not install a systemd unit.

Before deployment¶

Create and validate a configuration:

continuum-router config generate --template production-ha --output config.yaml
# Set required environment variables referenced by the template.
continuum-router config validate config.yaml

For a smaller starting point, use --template minimal. Available templates are minimal, development, multi-provider, production-ha, api-gateway, kv-cache-optimized, smart-routing, disaggregated, cost-optimized, sglang, and ab-testing.

Bind to loopback unless an external listener is required. Enable API and Admin authentication before exposing the service to an untrusted network. The router serves plain HTTP; terminate TLS in a trusted reverse proxy, ingress controller, or load balancer.

Release feature set¶

A default source build uses Cargo's full feature set. Official release binaries and official container images additionally compile control-plane, appproxy-router, and redis-cache; all three remain runtime opt-in. appproxy-legacy, bedrock-sigv4, and s3-cache remain source-build opt-ins unless a release workflow explicitly adds them.

redis-cache joined the official set so a multi-replica deployment can share rate-limit counters and the response cache without a custom build. Compiling it activates nothing: the router contacts Redis only when rate_limiting.storage: redis or response_cache.backend: redis names an endpoint. See Shared Redis state.

Runtime configuration cannot enable code that was not compiled. Check /admin/capabilities with Admin credentials when diagnosing a feature-gated deployment.

Docker¶

One multi-architecture image is published by the release workflow to GitHub Container Registry:

• ghcr.io/lablup/continuum-router:<version>: pinned version
• ghcr.io/lablup/continuum-router:<major>.<minor>: latest patch of that minor
• latest: latest non-prerelease release

Pin a version or digest in production instead of relying on latest.

The image is built on gcr.io/distroless/static-debian12 and carries the statically linked musl binary, a passwd file, and timezone data. There is no shell and no package manager. The base image also carries a CA bundle at /etc/ssl/certs/ca-certificates.crt and sets SSL_CERT_FILE to it. On Linux the router loads its trust roots from there, so outbound HTTPS to providers depends on that file. See Getting a shell for debugging a container that has no shell of its own.

docker run --rm \
  -p 8080:8080 \
  -v "$PWD/config.yaml:/etc/continuum-router/config.yaml:ro" \
  -e OPENAI_API_KEY \
  ghcr.io/lablup/continuum-router:2.0.0

The image runs as uid 65532 (the nonroot user distroless defines), starts with --config /etc/continuum-router/config.yaml, and uses the built-in --health-check command for its container health check.

Without a mounted file, that path holds the default configuration the image's own binary generated with --generate-config at build time (the same file as config.yaml.example). It binds 0.0.0.0:8080 and points at an Ollama backend on localhost:11434, so the container starts and serves /health but has no reachable backend until you mount your own configuration over it as shown above.

KV listener container¶

Images built from this Dockerfile.ci revision, and release images published after it, also contain continuum-kv-listener. Existing published images are not retroactively changed by this pull request. Run the listener with an entrypoint override when it is deployed separately from the router. The inherited router health check calls continuum-router --health-check, so disable it for a listener container and use the listener's /healthz endpoint in its orchestrator probes.

Set listen: "0.0.0.0:7817" in the listener configuration when the router reaches it over the deployment network; the example configuration binds loopback by default.

case "$(uname -s):$(uname -m)" in
  Linux:x86_64) asset_arch=x86_64 ;;
  Linux:aarch64|Linux:arm64) asset_arch=aarch64 ;;
  *) echo "This Linux image example supports only x86_64 and aarch64 hosts." >&2; exit 1 ;;
esac
asset="continuum-router-linux-${asset_arch}-musl.tar.gz"
mkdir -p docker-context
gh release download v1.30.0 --repo lablup/continuum-router \
  --pattern "$asset" --dir /tmp
tar -xzf "/tmp/$asset" -C docker-context
cp model-metadata.yaml docker-context/
cp -R model-metadata.d docker-context/
docker build --file Dockerfile.ci --build-arg VERSION=1.30.0 \
  --tag continuum-router-listener:local docker-context
docker run --rm --no-healthcheck \
  --entrypoint /usr/local/bin/continuum-kv-listener \
  -p 7817:7817 \
  -v "$PWD/config.kv-listener.yaml:/etc/continuum-router/config.kv-listener.yaml:ro" \
  continuum-router-listener:local \
  --config /etc/continuum-router/config.kv-listener.yaml

The archive selection keeps the router and listener from one matching Linux musl bundle and matches the image base architecture. The pinned v1.30.0 bundle predates packaged metadata catalogs, so the example copies the checkout's catalog into the Docker build context required by the current Dockerfile.ci. Run the commands on a supported Linux host; do not force an x86_64 archive into an ARM64 image.

For Kubernetes, replace the image entrypoint with command: ["/usr/local/bin/continuum-kv-listener"], pass the configuration path in args, and point both readiness and liveness probes at httpGet.path: /healthz on the listener port. The listener's SSE streams remain internal deployment traffic; restrict its ZMQ publisher and SSE network access as described in the KV-cache guide. When the router cannot reach the listener (for example the router runs outside the GPU cluster), enable the listener's event_bus push mode instead: the listener dials out to the router's TLS event bus, so it needs an outbound path to the router and no inbound one. See Push Mode.

Docker Compose¶

The repository includes docker-compose.yml. It expects ./config.yaml and supports a VERSION environment variable:

cp config.yaml.example config.yaml
# Replace the annotated sample with your real backend/auth values, then validate it.
continuum-router config validate config.yaml
VERSION=2.0.0 docker compose up -d
docker compose logs -f continuum-router

The Compose file uses ghcr.io/lablup/continuum-router:${VERSION:-latest}. Pass provider credentials through environment variables referenced by ${ENV_VAR} placeholders in the mounted config; do not hard-code credentials into the image or Compose file.

Dynamo prefill/decode fleets¶

A second Compose stack, deploy/examples/dynamo-direct-shim/, deploys the router in front of an NVIDIA Dynamo prefill/decode fleet. The router discovers the workers from etcd, reads each one's load over ZMQ, chooses a prefill and a decode worker per request, and dispatches through a dynamo.frontend --router-mode direct shim that honors the choice. It needs no GPU: the workers are dynamo.mocker processes, which register the same discovery records and publish the same kv_metrics as dynamo.vllm.

cd deploy/examples/dynamo-direct-shim
docker login nvcr.io          # optional: the pinned Dynamo image pulls anonymously
docker compose pull
docker compose up -d

Then confirm the fleet and the routing decision:

curl -s -H "Authorization: Bearer dynamo-e2e-local-admin-token" \
  http://127.0.0.1:8080/admin/dynamo/workers | python3 -m json.tool
curl -i -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' -H 'x-backend;' \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Say hi in one word."}],"max_tokens":16}'

The response's x-continuum-prefill-backend and x-continuum-decode-backend name two distinct dynamo:<instance_id> values when both phases were placed. They are written only because the request carries an x-backend header (the empty x-backend; above, which expresses no routing preference); without it the response carries x-continuum-routing-path alone.

Automation and Kubernetes deployment options for the Dynamo shim:

• End-to-end test script: scripts/e2e-dynamo-shim.sh runs the full flow and asserts every step, including degradation to aggregated selection once the prefill worker stops and an aggregated worker (aggregated compose profile) replaces the decode worker. Tear down a manually started stack with docker compose --profile aggregated down -v so the aggregated worker is removed too (.github/workflows/dynamo-e2e.yml runs the same script on manual dispatch).
• Kubernetes with external etcd: deploy/helm/continuum-router/values-dynamo-direct-shim.yaml and deploy/kubernetes/overlays/dynamo-direct-shim/ provide Helm and Kustomize profiles pointing at an external etcd endpoint and an existing shim Service.
• Kubernetes Pod discovery (kubernetes_pods, no etcd): When the router only has permission to read Pods in a namespace, set dynamoPodDiscovery.enabled: true (and dynamoPodDiscovery.namespace) in Helm, or apply deploy/kubernetes/overlays/dynamo-pod-discovery/ to grant get/list/watch on pods; see Dynamo Worker Discovery.

The architecture, header contract, selection formulas, compatibility table, and the explicit list of what stays Dynamo-owned are in Dynamo integration.

Build repository Dockerfiles¶

Dockerfile downloads the matching GitHub release archive. Supply the release version explicitly:

docker build --build-arg VERSION=2.0.0 -t continuum-router:2.0.0 .

It is not a source-build Dockerfile. For a customized feature set, compile the binary separately and construct an image that copies that artifact, or adapt Dockerfile.ci with an intentional build pipeline.

Kubernetes¶

The maintained Kustomize bundle is in deploy/kubernetes/base. It includes three replicas, resource requests and limits, startup/readiness/liveness probes, a ClusterIP Service, TLS Ingress, HPA, PodDisruptionBudget, restricted pod security settings, a read-only root filesystem, and ingress/egress NetworkPolicy. The staging overlay in deploy/kubernetes/overlays/staging is used by the automated staging workflow.

Review every operator-specific value before applying it: provider/model configuration, the image tag or digest, Ingress host and TLS Secret, NetworkPolicy namespace selectors, egress rules, resource sizing, and trusted proxies. Create continuum-router-secrets separately; the checked-in bundle never creates or stores credentials.

kubectl apply -f deploy/kubernetes/base/namespace.yaml
kubectl -n continuum-router create secret generic continuum-router-secrets \
  --from-literal=OPENAI_API_KEY="$OPENAI_API_KEY" \
  --from-literal=ROUTER_API_KEY="$ROUTER_API_KEY" \
  --from-literal=ADMIN_TOKEN="$ADMIN_TOKEN"
kubectl apply -k deploy/kubernetes/base
kubectl -n continuum-router rollout status deployment/continuum-router --timeout=5m

Run scripts/validate-deployment-assets.sh after changing the manifests. It renders the base, staging overlay, monitoring bundle, all Helm profiles, and the event broker chart and bundle; CI also validates them.

The following minimal example remains useful as an explanation of the core objects. The files under deploy/kubernetes/ are the maintained source of truth for deployment:

apiVersion: v1
kind: ConfigMap
metadata:
  name: continuum-router-config
data:
  config.yaml: |
    server:
      bind_address: 0.0.0.0:8080
      workers: 4
    selection_strategy: RoundRobin
    backends:
      - name: ollama
        type: ollama
        url: http://ollama:11434
        models: [llama3.2]
    health_checks:
      interval: 30s
      timeout: 5s
      unhealthy_threshold: 3
      healthy_threshold: 2
      endpoint: /health
    logging:
      level: info
      format: json
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: continuum-router
spec:
  replicas: 2
  selector:
    matchLabels:
      app: continuum-router
  template:
    metadata:
      labels:
        app: continuum-router
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: router
          image: ghcr.io/lablup/continuum-router:2.0.0
          args: ["--config", "/etc/continuum-router/config.yaml"]
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
          livenessProbe:
            httpGet:
              path: /health
              port: http
          volumeMounts:
            - name: config
              mountPath: /etc/continuum-router/config.yaml
              subPath: config.yaml
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              # Size this together with server.max_concurrent_requests, not
              # independently. See "Capacity planning: memory" below:
              # worst-case resident memory is concurrency times the
              # per-request budget. File uploads stream to disk and no longer
              # dominate that budget, but a request at the file-resolution
              # ceiling still contributes about 43 MiB once base64-expanded.
              memory: 1Gi
      volumes:
        - name: config
          configMap:
            name: continuum-router-config
---
apiVersion: v1
kind: Service
metadata:
  name: continuum-router
spec:
  selector:
    app: continuum-router
  ports:
    - name: http
      port: 8080
      targetPort: http

Use a Secret manager or Kubernetes Secret-to-environment integration for provider credentials. The ${ENV_VAR} expansion happens inside the router process.

/health is the unauthenticated liveness endpoint. /health/ready is the unauthenticated serving-pool readiness endpoint; it returns 503 until at least one enabled, non-internal backend is active. /healthz does not exist. /version is also public. Metrics use the configured metrics path and their own authentication settings, and are served on the dedicated metrics listener at metrics.port (the metrics container port 9090, which the Service, the scrape annotations, and the NetworkPolicy's monitoring rule target) as well as on the API port.

The default readinessProbe above has no explicit periodSeconds/failureThreshold. It uses /health/ready, so a replica with an empty serving pool is removed from load balancing while /health continues to prove that the process is alive. By default (health_checks.block_startup: true), listener binding waits for connection pre-warm and the initial health-check round across every backend. Set health_checks.block_startup: false to bind listeners first and let readiness follow serving-pool reconciliation while backend health converges in the background; see Startup Behavior for details.

A replica whose local backends: is empty and that waits for its first Continuum Hub assignment stays unready until that assignment reaches the serving pool. With maxUnavailable: 0 a rollout pauses on it until the Hub delivers, which is the intended behavior rather than a stuck probe. A replica that restarts with a persisted Hub backend snapshot becomes ready once the restored backends are rebuilt into the pool, without waiting for a new Hub revision.

Shutdown and rolling updates¶

The HTTP server handles SIGINT/SIGTERM and starts graceful shutdown. Every listener stops accepting new connections at once and idle keep-alive connections close, so a load balancer sees the replica refuse connections and sends new traffic elsewhere. Requests already in progress, streaming responses included, keep running until they finish or server.shutdown_drain_timeout passes (default 20s); whatever is still open at the deadline is aborted. A second signal during the drain aborts at once. Background services such as the control-plane usage flush, stats persistence, and OTLP export stop after the drain, so the usage of drained requests is still delivered.

Keep shutdown_drain_timeout plus those background waits inside the orchestrator grace period, or the process is killed mid-cleanup. Without the control-plane agent and OTLP export the background waits take at most about 8 seconds, which the 20s default leaves room for inside the Kubernetes default terminationGracePeriodSeconds: 30. With the control-plane agent enabled, its usage flush may wait up to its hub request timeout plus 5 seconds (15 seconds with defaults), and OTLP export adds up to tracing.otlp.shutdown_timeout (5s); in that case set the grace period to at least the drain timeout plus 25 seconds. When long streams must survive a rollout, raise both together:

server:
  shutdown_drain_timeout: 90s   # pair with terminationGracePeriodSeconds: 120

Kubernetes keeps routing to a pod for a few seconds after it sends SIGTERM, while the endpoint removal propagates, and those connections are refused once the listener closes. A preStop hook such as sleep 5 delays the signal until the endpoint is gone; add its duration to the grace period too.

The value is read at startup, so a change takes effect from the next process. A WebSocket session (the realtime proxy) is not drained: once upgraded it ends when the process exits.

Run at least two replicas for rollout availability. Router-local state such as in-memory rate limits, response cache, stored Responses sessions, and some statistics is not automatically shared across replicas. Select Redis/persistent features where supported or accept per-instance behavior.

Helm¶

deploy/helm/continuum-router provides development, staging, canary, and production values profiles. The chart renders the same security and availability controls as the maintained Kustomize bundle and can optionally create a Prometheus Operator ServiceMonitor.

helm lint deploy/helm/continuum-router
helm template router deploy/helm/continuum-router \
  --namespace continuum-router \
  -f deploy/helm/continuum-router/values-production.yaml
kubectl create namespace continuum-router --dry-run=client -o yaml | kubectl apply -f -
kubectl label namespace continuum-router app.kubernetes.io/name=continuum-router --overwrite
helm upgrade --install router deploy/helm/continuum-router \
  --namespace continuum-router \
  -f deploy/helm/continuum-router/values-production.yaml \
  --set-string image.digest="$IMAGE_DIGEST"

Production installs should set IMAGE_DIGEST to an approved sha256:... manifest digest, replace example hosts, provide existingSecret, and review the default NetworkPolicy. The chart defaults to per-replica in-memory rate limits. Layer values-shared-redis.yaml on top of a profile to switch to shared state; see Shared Redis state.

Automated staging deployment¶

.github/workflows/deploy-staging.yml deploys a successful published-release workflow run, or a manually selected image tag, to the continuum-router-staging namespace. Configure the protected GitHub staging environment with KUBECONFIG_B64, ROUTER_HOST, required reviewers, and deployment-branch restrictions. Create the continuum-router-staging namespace first, label it app.kubernetes.io/name=continuum-router, and then create continuum-router-secrets and continuum-router-tls there before the first run.

The workflow accepts only published semver releases, verifies the release manifest attestation, resolves the image to an immutable digest, uses a single non-cancelling concurrency group, performs a server-side apply, and waits for the Deployment rollout. All high/critical image vulnerabilities block publication, and the build publishes SBOM and provenance attestations for the final multi-architecture manifests.

Deployment strategies¶

The maintained Deployment uses a rolling update with maxUnavailable: 0; use helm rollback or reapply the previous immutable image for rollback. For blue-green deployment, install the candidate as a separate release, verify its health and metrics, then switch the ingress or upstream Service atomically while retaining the previous release during the observation window.

For a canary, install a second release with values-canary.yaml and an immutable candidate image. It creates one labeled replica without a separate Ingress, HPA, or disruption budget. Use the ingress controller or service mesh to assign a small explicit weight to that release's Service, observe error rate and latency, and promote it by upgrading the primary release. The chart does not assume a vendor-specific weighted-routing annotation. Configuration and feature rollouts can use the same two-release pattern; account for router-local sessions, caches, statistics, and rate-limit counters while releases overlap.

Automated progressive delivery with Argo Rollouts¶

The strategies above are controller-independent and stay that way: the base chart, the Kustomize bundle, values-canary.yaml, and the blue-green two-release switch all work with nothing installed beyond Kubernetes. What they do not give you is automation. Traffic shifting, metric analysis, promotion, and rollback are all manual, which is fine for a planned release and poor at 3 a.m.

The chart can hand that work to Argo Rollouts. It is off by default and gated on one value.

Enabling it¶

The controller and its CRDs must exist in the cluster first. Without them the API server rejects the Rollout object:

kubectl get crd rollouts.argoproj.io analysistemplates.argoproj.io

Then layer the overlay on an environment profile:

helm upgrade --install router deploy/helm/continuum-router \
  --namespace continuum-router \
  -f deploy/helm/continuum-router/values-production.yaml \
  -f deploy/helm/continuum-router/values-progressive-delivery.yaml \
  --set-string image.digest="$IMAGE_DIGEST"

What changes in the rendered manifests:

The two workload objects share one pod template (continuum-router.podTemplate in _helpers.tpl), so a security context, probe, or volume added to one is added to the other. They cannot drift.

The HPA retarget matters: an HPA still pointing at a Deployment that no longer exists sits inert and the router never scales. Pointed at the Rollout, Argo Rollouts consumes the desired replica count and distributes it across the stable and canary ReplicaSets itself.

Weighted canary steps¶

progressiveDelivery.argoRollouts.steps is passed to strategy.canary.steps verbatim, so any step Argo Rollouts understands is available. The production overlay ships 10%, 30%, 60% with a ten-minute soak between each.

Without a traffic-routing provider the weight is approximated by ReplicaSet pod counts behind the single Service. That is accurate enough to observe a canary and is the most the generic chart can guarantee, since it assumes no particular mesh or ingress. For enforced weighting, set progressiveDelivery.argoRollouts.trafficRouting.enabled: true and put the provider block in trafficRouting.spec; the chart then renders <release>-canary and <release>-stable Services and passes the block through:

progressiveDelivery:
  argoRollouts:
    trafficRouting:
      enabled: true
      spec:
        nginx:
          stableIngress: continuum-router

Analysis¶

AnalysisTemplate <release>-health runs two measurements against the router's own metrics, scoped to the canary ReplicaSet:

• error-rate: sum(rate(errors_total[2m])) / sum(rate(http_requests_total[2m])), must stay at or below analysis.maxErrorRate (0.05).
• latency-p95: histogram_quantile(0.95, ...http_request_duration_seconds_bucket...), must stay at or below analysis.maxLatencyP95Seconds (2.0).

Scoping to the canary is what makes the measurement meaningful, and it depends on one label. The chart's ServiceMonitor relabels rollouts-pod-template-hash from the pod onto every series as rollouts_pod_template_hash, and the standalone bundle in monitoring/prometheus/ carries the same rule. A Prometheus configured elsewhere needs it too, or the analysis measures the fleet average and a bad canary hides inside good stable traffic:

relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_rollouts_pod_template_hash]
    target_label: rollouts_pod_template_hash

Both queries end with and sum(rate(http_requests_total{...})) > analysis.minRequestRate. Below that rate the query returns an empty vector, neither the success nor the failure condition matches, and Argo Rollouts records the measurement as Inconclusive. An inconclusive run pauses for an operator rather than deciding. This is the low-traffic guard, and it is the difference between a canary judged on evidence and one judged on three overnight requests: a single failure out of three is a 33% error rate, which would roll back a perfectly good release. analysis.inconclusiveLimit (3) bounds how long it waits before pausing.

Analysis runs in the background from analysis.startingStep onward, not only between steps, so a canary that degrades mid-soak aborts immediately instead of at the next step boundary. The default 1 is a zero-based index into steps, which puts the first measurement in the first soak, after the first weight has been applied and traffic is actually reaching the canary. Starting at 0 would measure before any weight is set, where there is nothing to measure.

Tuning, in the order worth trying:

Promotion¶

With manualPromotionPause: true (the production overlay) the rollout stops after the last weighted step and waits indefinitely. Promotion is a deliberate act:

kubectl argo rollouts get rollout router-continuum-router --watch
kubectl argo rollouts promote router-continuum-router

The pause blocks promotion only. Analysis keeps running through it, so a canary that degrades while waiting for approval still aborts on its own. Set manualPromotionPause: false where an unattended promotion is acceptable, for example in staging.

Rollback¶

Failed analysis aborts the rollout automatically: the canary ReplicaSet scales to zero and the stable ReplicaSet keeps serving, which means the previously running version was never taken out of service. Manually:

kubectl argo rollouts abort router-continuum-router     # stop and return traffic to stable
kubectl argo rollouts undo router-continuum-router      # roll back to the previous revision

Drill the failure path before you depend on it. On a staging install, deploy an image that fails readiness or returns errors, watch the canary weight stop advancing, and confirm with kubectl argo rollouts get rollout <name> that the analysis run is Failed, the rollout is Degraded, and the stable ReplicaSet still has its full replica count. That last check is the one that matters: an abort that also scaled down stable would be an outage, not a rollback.

Configuration rollouts¶

A configuration change is delivered by the same mechanism, and it needs no extra setting. The pod template carries a checksum/config annotation over the rendered ConfigMap, so editing routerConfig (or the ConfigMap named by existingConfigMap) changes the pod template hash, which starts a fresh canary progression through the same steps and the same analysis. A bad limit, a bad timeout, or a bad backend inventory is caught on 10% of traffic and rolled back automatically, exactly like a bad image.

Two things this does not cover. Settings the router hot-reloads in place are applied by the running process without a new pod, so they bypass the rollout entirely; deliver those through a config revision (a ConfigMap change) when you want them gated. And Cargo features are compile-time, so "feature rollout" always means rolling out a configuration revision or an image, never flipping a runtime flag; see Release feature set.

Recovering to a plain Deployment¶

Progressive delivery is reversible. Reinstall without the overlay:

helm upgrade --install router deploy/helm/continuum-router \
  --namespace continuum-router \
  -f deploy/helm/continuum-router/values-production.yaml \
  --set-string image.digest="$IMAGE_DIGEST"
kubectl -n continuum-router delete rollout router-continuum-router
kubectl -n continuum-router rollout status deployment/router-continuum-router --timeout=5m

Delete the Rollout after the Deployment is up, not before: the order keeps pods serving throughout. If the controller itself is broken or being uninstalled, this is also the escape hatch, and it needs nothing from the controller to work.

Flagger¶

The chart does not ship Flagger resources, but the mapping is direct if that is your ecosystem: a Canary with analysis.steps replaces the Rollout, analysis.metrics with a MetricTemplate replaces the AnalysisTemplate, and the same two PromQL queries and the same low-traffic guard apply unchanged. Flagger keeps the Deployment and generates its own ReplicaSets, so the deployment guard here is not needed; leave progressiveDelivery.argoRollouts.enabled: false and install the Canary alongside the chart.

In-cluster event broker¶

A router that cannot dial the Dynamo workers' Pod IPs, typically one running outside the GPU cluster, receives no kv_metrics load and no kv-events from them. The event broker tunnel reverses the direction of that connection: continuum-event-broker runs inside the cluster, dials out to a TLS listener on the router (the event bus), and relays the workers' ZMQ publishers through it. The protocol, the broker's security model, and its metrics are in In-Cluster Event Broker. Deploying the tunnel has two halves, one in each cluster.

  router cluster                                  GPU cluster (fleet namespace)
  ┌──────────────────────────────┐   TLS, h2    ┌──────────────────────────────┐   ZMQ    ┌────────┐
  │ continuum-router             │◀─────────────│ continuum-event-broker a, b  │─────────▶│ worker │
  │ dynamo.event_bus :7443       │  dialed out  │ (one Pod per broker id)      │          │ pubs   │
  │ Service <release>-event-bus  │  by broker   │ reads DynamoWorkerMetadata   │          └────────┘
  └──────────────────────────────┘              └──────────────────────────────┘

Router side: the bus listener¶

The listener is the dynamo.event_bus section of the router configuration, built into official release binaries through the dynamo-event-bus feature and off by default. It is its own port, never the API's, speaks TLS 1.3 with ALPN h2 only, and refuses plaintext on the first byte. The Helm chart exposes it behind dynamoEventBus; values-dynamo-event-bus.yaml is the complete overlay:

kubectl -n continuum-router create secret tls continuum-router-event-bus-tls \
  --cert=bus.crt --key=bus.key
helm upgrade --install router deploy/helm/continuum-router \
  --namespace continuum-router \
  -f deploy/helm/continuum-router/values-production.yaml \
  -f deploy/helm/continuum-router/values-dynamo-event-bus.yaml \
  --set-string image.digest="$IMAGE_DIGEST"

When dynamoEventBus.enabled is set:

• Rendered resources: The chart adds the event-bus container port, mounts the kubernetes.io/tls Secret named by dynamoEventBus.tlsSecret (and the optional client CA from clientCASecret) at dynamoEventBus.mountPath, renders a separate <release>-event-bus Service, and admits the port in the NetworkPolicy. Enabling it without a TLS Secret fails the render because the chart never generates certificates.
• Router configuration alignment: routerConfig must match the chart mounts (dynamo.event_bus.listen on dynamoEventBus.port, and tls.cert/tls.key under mountPath). The mount path is /var/run/secrets/continuum-router/event-bus rather than under /etc/continuum-router, which is a read-only config mount that cannot hold a nested mount; values-dynamo-event-bus.yaml keeps both sides aligned in one file.
• Kustomize deployments: The Kustomize bundle does not include a prebuilt router-side event-bus overlay; add the same port, Secret mount, Service, and NetworkPolicy ingress rule to your overlay manually.

Router side: the broker credentials¶

Each broker identity has its own bearer token, 32 to 512 bytes. The token lives only in a Secret in the GPU cluster; the router is configured with its SHA-256 alone, so the router's configuration never holds a usable credential:

TOKEN=$(openssl rand -hex 32)
kubectl -n dynamo create secret generic continuum-event-broker-a --from-literal=credential="$TOKEN"
printf %s "$TOKEN" | sha256sum    # -> dynamo.event_bus.brokers[].token_hash: "sha256:<digest>"

List every identity under dynamo.event_bus.brokers with its id (the broker's client_id), its token_hash, and its namespaces, the Dynamo namespaces it may declare, which must cover the broker's dynamoNamespaces. Never leave namespaces empty: an empty allowlist lets any valid token declare any namespace. Removing an entry or changing its hash reloads live and ends that broker's session with credential_revoked. With dynamo.event_bus.tls.client_ca set, a client certificate whose DNS SAN is the broker id authenticates on its own, and token_hash becomes optional.

The router keeps one session per broker id: a second session with the same id supersedes the first. That is why high availability comes from two identities rather than from two replicas of one: two Pods sharing an id would take each other's session in a loop.

On the router side, each broker session lands on one router Pod (whichever Pod the <release>-event-bus Service selected when the broker dialed):

• Within a single router Pod: Multiple broker sessions may cover the same Dynamo namespace. Subscriptions distribute across covering brokers via stable rendezvous ranking, and if a broker disconnects, its subscriptions fail over to the next covering broker within one reconnect.
• Across multiple router replicas: Router replicas do not share or coordinate broker sessions. A replica with no covering broker session keeps its subscriptions pending and treats the workers it cannot hear as having no fresh load sample, just as when any load sample is missing.
• Sizing broker identities: Either run the tunnel-serving router as a single replica, or deploy at least as many broker identities as router replicas (noting that the Kubernetes Service, not the router, determines which replica each broker reaches). Inspect per-replica coverage via GET /admin/dynamo/workers (broker_coverage) and continuum_dynamo_broker_namespace_sessions{namespace}.

Router side: reachability from the cluster¶

The brokers connect to the router, never the other way, so the only path that has to exist is from the GPU cluster's Pods to the bus address. Check it before relying on the tunnel:

• Expose the listener where the brokers can reach it. When they run in another cluster, set dynamoEventBus.service.type: LoadBalancer (the overlay does) or put a TLS passthrough ingress in front of a ClusterIP Service. The listener terminates TLS and negotiates HTTP/2 itself, so a TLS-terminating proxy in front of it breaks the protocol.
• The bus certificate must carry the name the brokers dial: their bus.url host, or bus.tlsServerName when that differs. Give the brokers the issuing CA through bus.caBundle; without one they trust only the public web roots.
• Restrict the sources: dynamoEventBus.service.loadBalancerSourceRanges at the load balancer and dynamoEventBus.networkPolicy.fromCidrs in the router's NetworkPolicy, both set to the GPU cluster's egress addresses as the router's cluster sees them. With neither, the port is open to any source and only TLS and the broker credential protect it.
• On the GPU cluster side, the broker's own NetworkPolicy must allow egress to that address and port (networkPolicy.bus.cidrs and ports).
• Confirm from a Pod in the fleet's namespace that the address resolves and that a TLS handshake with ALPN h2 completes, for example openssl s_client -connect router-bus.example.com:7443 -alpn h2 -servername router-bus.example.com. The broker's continuum_event_broker_session_connected gauge reads 1 once the router has welcomed its session, and the router's continuum_dynamo_event_bus_sessions counts the welcomed sessions.

GPU cluster side: the broker¶

deploy/helm/continuum-event-broker is a chart of its own, not an option of the router chart, because the broker is installed in a different cluster and namespace than the router, from a different image (ghcr.io/lablup/continuum-event-broker), with different RBAC, on its own release cycle. The router chart deploys no Dynamo component for the same reason, and continuum-kv-listener is not an option of it either. deploy/kubernetes/event-broker/ is the equivalent standalone Kustomize bundle, applied to the GPU cluster rather than layered on deploy/kubernetes/base.

kubectl -n dynamo create configmap continuum-router-event-bus-ca --from-file=ca.crt=router-event-bus-ca.pem
helm upgrade --install event-broker deploy/helm/continuum-event-broker \
  --namespace dynamo \
  -f my-broker-values.yaml          # start from values-example.yaml
# or: kubectl apply -k deploy/kubernetes/event-broker

Both render:

• One Deployment of exactly one Pod per brokers[] entry (two in the example), each with its own ConfigMap (client_id) and credential Secret, preferably on different nodes, and a PodDisruptionBudget that keeps one of them through a node drain. The chart refuses a missing bus.url, a plaintext http:// bus, a broker without credentialSecret.name, and an id listed twice.
• A ServiceAccount whose token is mounted, and a Role and RoleBinding granting get, list, and watch on dynamoworkermetadatas.nvidia.com, and nothing else, in each watched namespace (the release namespace unless watchNamespaces lists others). No cluster-scoped grant.
• The credential mounted from the existing Secret at /var/run/secrets/continuum-event-broker/credential, and the router's CA bundle from an existing ConfigMap or Secret, or from inline bus.caBundle.pem. No Secret is ever rendered.
• Readiness on /readyz (200 once every watched namespace has listed) and liveness on /healthz, on the metrics port 9464. Neither depends on the bus session, so a router outage does not restart the broker; it keeps redialing with backoff.
• A NetworkPolicy that denies all other traffic:
  • Egress: Allowed only to DNS, the Kubernetes API server, the router's bus address (networkPolicy.bus), and worker publishers (networkPolicy.publishers, which defaults to watched namespaces and allowedCidrs on any port; narrow the ports to those your workers publish on).
  • Ingress: Allowed only to the metrics port from networkPolicy.monitoringNamespace.
  • Empty-list safety: Every rule renders only when it names a destination or port; emptying a list removes its rule rather than rendering an allow-all rule.

To pull the image from a private registry or mirror, set image.repository and name an existing kubernetes.io/dockerconfigjson Secret (kubectl create secret docker-registry) in imagePullSecrets, as - name: <secret> or - <secret>. The chart adds it to every broker Pod and never renders the credential; it refuses an entry of another shape, an invalid Secret name, and a Secret listed twice. The Kustomize bundle takes the same Secret through a patch, shown in its kustomization.yaml.

Set allowedCidrs to the cluster's Pod CIDR. The DynamoWorkerMetadata check trusts whoever can create that resource in a watched namespace, and the allow-list bounds what a forged advertisement can make the broker dial. Each broker Deployment rolls with a one-Pod surge: for the few seconds the old and new Pod of one identity both run they supersede each other's session, which both ends recover from, where Recreate would leave that identity uncovered for a whole drain and start.

scripts/validate-deployment-assets.sh lints and renders the broker chart and bundle and checks this contract: exactly the metadata read in exactly the watched namespaces, one Pod per distinct broker id, the credential only from a Secret, both probes, no allow-all egress rule with every egress list emptied, and the refusals above. It also renders the router overlay and checks that the listener port, the TLS mount, and the configured paths agree.

systemd¶

The Debian package installs the binary, examples, documentation, and manpage but no unit file. Create a local unit if systemd management is desired:

[Unit]
Description=Continuum Router
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=continuum-router
Group=continuum-router
EnvironmentFile=-/etc/continuum-router/environment
ExecStart=/usr/bin/continuum-router --config /etc/continuum-router/config.yaml
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/continuum-router /var/log/continuum-router
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Create the user and writable paths to match the features you enable, then validate before starting:

sudo /usr/bin/continuum-router config validate /etc/continuum-router/config.yaml
sudo systemctl daemon-reload
sudo systemctl enable --now continuum-router
journalctl -u continuum-router -f

For a tarball installation, adjust ExecStart to /usr/local/bin/continuum-router. Tighten or expand sandbox paths based on Files API, stats persistence, OAuth token stores, control-plane state, Unix sockets, and cache storage.

Capacity planning: memory¶

Worst-case resident memory is a product, not a sum:

worst-case memory  ~=  (concurrent requests)  x  (per-request budget)

Every memory bound the router enforces is on the right-hand factor. Nothing bounded the left one until server.max_concurrent_requests, so before you set it the ceiling is whatever your clients can open.

The per-request terms¶

Each is an independent ceiling on one request. A single request does not hit all of them at once, but each is reachable on its own.

• How the streaming queue and process budget work: Every streaming relay reads the upstream at the upstream's pace so that a slow reader disconnecting before the end cannot escape usage accounting. Each stream's relay queue is capped at 256 MiB, and all active SseParser buffers, SseLineBuffer buffers, and relay queues share the process-wide server.max_stream_buffer_bytes budget (default 4 GiB; 0 is rejected at parse time, and changing it requires a restart). When a slow reader reaches the 256 MiB queue ceiling or exhausts the process budget, the router logs a warning, drops further client frames without pacing the upstream reader (always admitting one frame if the queue is empty), omits the trailing data: [DONE] sentinel so a truncated stream cannot appear complete, and continues reading the upstream to record terminal usage. An SseParser buffer that exceeds server.max_stream_buffer_bytes stops the stream with a size-limit error. Pair server.max_concurrent_requests and server.max_stream_buffer_bytes with an inbound idle timeout at your proxy or ingress.

Doing the multiplication¶

A rate limit does not supply the concurrency factor. It bounds arrivals per unit time, not the number resident at once. By Little's law:

in-flight  ~=  arrival rate  x  mean request duration

LLM completions routinely run tens of seconds, so per_client: 10 requests per second at a 30 second mean duration implies roughly 300 concurrent requests from one client, entirely within policy. At the file-resolution ceiling that is about 12.6 GiB resident; at body sizes alone, about 2.9 GiB before counting parsed JSON. Uploads do not contribute to that figure in memory, but ten concurrent uploads at the default files.max_file_size do occupy 5 GiB of disk until they finish, so size files.storage_path against the concurrency you allow.

Setting the ceiling¶

server:
  max_concurrent_requests: 64

Unset by default, which means no ceiling. Over the ceiling the router responds 503 with a Retry-After and a JSON body naming the limit (the Anthropic overloaded_error envelope on /anthropic/* routes), rather than queueing: queueing would trade an out-of-memory kill for unbounded latency while still holding the memory.

Size it from the memory you are willing to spend, not from CPU:

max_concurrent_requests  ~=  (container memory limit - baseline)  /  (expected per-request cost)

Two things to know before choosing a number:

• A permit is held for the whole request, which for a streaming completion is the whole stream. A ceiling tuned against short non-streaming traffic will starve long streaming traffic. Size it against your longest requests.
• /health, /health/ready, /healthz, and the configured metrics.path are exempt, so a saturated router still answers its own liveness and readiness probes and stays scrapeable. Shedding the probe is how a correctly-degrading replica gets restarted in the middle of the overload it is handling.

Pair the ceiling with rate_limiting rather than choosing between them: the rate limit bounds arrival, the ceiling bounds residency, and neither substitutes for the other. A reverse proxy in front helps only if it is explicitly configured to cap concurrent connections and body size, and it cannot know the per-request cost of a request carrying 100 file references.

Pair it with a request timeout as well. The router applies timeouts to its upstream calls, not to how long an inbound client may hold a slot. A client that opens a streaming completion and then stops reading pins its permit for as long as the connection lives, so a ceiling on its own converts a memory-exhaustion attack (expensive for the attacker) into a slot-exhaustion one (cheap: N idle sockets). Enforce an inbound request or idle timeout at your proxy or ingress.

One reporting caveat if you consume the control-plane supply feed: max_concurrency is the ceiling across every route, while active_requests counts only the API routes. A router saturated by file uploads therefore shows a low active_requests against a full ceiling, so do not read the ratio as utilization.

Sizing the container¶

The Kubernetes example above requests 256Mi and limits 1Gi. Treat those numbers as a floor for an idle router, and raise both together with the ceiling you set. One request at the file-resolution ceiling still contributes about 43 MiB once base64-expanded, so the concurrency ceiling and the container memory limit are one decision. For example, max_concurrent_requests: 64 with an expected 10 MiB per request wants roughly 1Gi of headroom above baseline.

files.max_file_size is primarily a disk budget rather than a memory budget:

• Disk sizing and startup check: Size files.storage_path to at least files.max_file_size times allowed upload concurrency. The router warns at startup if files.max_file_size is large relative to the container memory limit, because a few consumers (base64 file injection into chat requests and provider passthrough re-upload) still load a whole file in memory.
• Chat base64 injection caps: The chat-completions base64 injection path checks stored metadata before reading content and refuses individual files above 10 MiB or requests exceeding 32 MiB raw total, so larger uploads on disk cannot be expanded into a single chat request.

Under an orchestrator with a memory limit set, exceeding it is an OOM kill and a restart: the blast radius is that replica's availability plus every in-flight request on it. On bare metal, or in a container with no memory limit, it is host-level pressure instead.

Capacity planning: throughput¶

The section above bounds what one replica may hold. This one is about what it can serve, and about the shipped defaults that depend on that: the 200m CPU request and 1 core limit in deploy/kubernetes/base/deployment.yaml, and the 70% CPU target in the HorizontalPodAutoscaler.

Measure it, do not read it off a page¶

Router throughput is not a constant. It is a function of your upstream provider latency, your streaming mix, the request shape, and the CPU you give a replica, and only the last of those is under this project's control. A headline requests-per-second figure published here would be wrong for almost every deployment. So the repository ships the measurement instead:

perf/run.sh smoke      # three seconds, confirms the harness works
perf/run.sh            # both capacity profiles, three trials each, then the gate

Nothing external is involved. The harness starts its own mock backends with a configured first-token delay and inter-token cadence, generates a router configuration pointing at them, starts the router as a separate process, drives sustained load over real sockets, and writes JSON containing throughput, latency percentiles, time to first token, error rate, router process RSS and CPU, and backend-pool occupancy.

Two profiles ship: non-streaming (32 connections, upstream fixed at 60ms) and streaming (32 connections, 64 tokens at 5ms). perf/README.md documents the profiles and the result schema. .github/workflows/perf.yml runs the same profiles weekly against reviewed thresholds in perf/thresholds.json, on a pinned runner class, and never on a per-commit build.

Run it on hardware matching your target replica, with the request shape you actually serve. A measurement on a workstation tells you nothing about a 200m CPU request.

Turning a measurement into a replica count¶

Two facts do most of the sizing work.

Concurrency, not request rate, sets the replica count. By Little's law, in-flight ~= arrival rate x mean duration. A completion that runs 20 seconds at 5 requests per second is 100 concurrent requests, and it is those 100 slots a replica has to hold, not the 5 per second. Read throughput_rps from the non-streaming profile as the ceiling on the router's own request-path work at the offered concurrency, and treat load above it as needing another replica rather than a bigger one.

The router relays streams rather than buffering them. The streaming profile measures this directly, and it is the reason a streaming replica is sized differently: with a backend that emits its first token at 50ms and finishes the stream about 315ms later, the measured time to first token stays near the upstream first-token delay rather than near the full stream duration. So the cost of a streaming replica is dominated by how many streams are open at once, and each open stream holds its max_concurrent_requests permit for the stream's entire life. Size against concurrent streams and their duration, not against completed requests per second.

The two calculations are separate and both are needed:

• Throughput tells you how many replicas. Divide your expected peak concurrency by the concurrency at which you measured.
• Memory tells you how large each replica is. Use the previous section, not this one, to choose server.max_concurrent_requests and the container memory limit.

Validating the autoscaling defaults on a cluster¶

The bundle ships minReplicas: 3 and maxReplicas: 10 (20 in the Helm production values) against a 70% CPU utilization target. Those defaults assume a replica becomes CPU-bound before it becomes memory-bound. Whether that holds is a property of your workload, and it is worth confirming before you rely on it:

1. Establish a per-replica baseline inside the cluster. Run the single-replica profile from a Job on the same node class as the router, so the measurement inherits the cluster's CPU limits rather than a laptop's.
2. Scale the offered load. Point the driver at the Service with continuum-perf run --target-url <service-host>:<port>, and raise offered concurrency past one replica's measured ceiling. In --target-url mode the harness cannot read /proc for an in-cluster process, so take memory and CPU from kubectl top pod instead.
3. Watch the autoscaler and the latency together. kubectl get hpa continuum-router --watch alongside the driver's p95. The defaults hold if replicas rise before p95 leaves the single-replica band, if the 30 second scale-up stabilization window is short enough for latency to recover inside it, and if the 300 second scale-down window does not thrash under sawtooth load.
4. Check the failure mode that matters. If CPU utilization stays well under 70% while latency climbs, the replica is not CPU-bound at your workload shape, and a CPU-target HPA will not scale it at all. That is the common case for a streaming-heavy deployment, where a replica is bound by open streams rather than by compute. Lower averageUtilization, or scale on a custom metric such as in-flight requests, rather than raising maxReplicas and hoping.

Record what you measure. A capacity number without the request shape, the upstream latency, and the replica size that produced it is not reproducible, which is exactly why the harness writes all three into every result file.

High availability¶

Continuum Router has no regions:, geographic_routing:, failover:, or postgresql: top-level configuration. Implement cross-region traffic management and TLS with external infrastructure.

Within one router instance, health filtering, circuit breaking, retry, model fallback, and the six selection strategies provide backend-level resilience. Ordinary proxy traffic records outcomes and routes around open backend circuits. Across router replicas:

• Use an external load balancer with /health/ready readiness checks and /health liveness checks.
• Keep configuration and secret versions consistent.
• Understand which rate-limit/cache/statistics stores are per process.
• Avoid assuming stored /v1/responses/{id} sessions can be retrieved from another replica without sticky ingress; current session storage is process-local.
• Plan Files API storage and ownership metadata for shared access before scaling it horizontally.

Shared Redis state¶

Rate-limit counters and the response cache are per process by default, so a three-replica deployment admits three times the configured rate and caches the same response three times. Pointing every replica at one Redis or Valkey endpoint removes both. Official binaries and images compile redis-cache, so this needs configuration only.

Supported topology¶

The boundary follows from the client: the router builds one deadpool-redis pool over redis::aio::ConnectionManager, which discovers no Sentinel master and routes no cluster slots. A Sentinel or Cluster URL would either fail to connect or, worse, pin every replica to whichever single node answered. Rather than imply support, the configuration loader refuses those URLs with a message naming the limitation, at startup, at hot reload, through the admin config API, and in continuum-router config validate.

Configuration¶

rate_limiting:
  enabled: true
  storage: redis
  redis:
    url: "${REDIS_URL}"
    key_prefix: "continuum:ratelimit:"
    ttl: 3600

response_cache:
  enabled: true
  backend: redis
  ttl: "300s"
  redis:
    url: "${REDIS_URL}"
    pool_size: 8
    key_prefix: "cr:resp:"
    fallback_to_memory: true

TLS comes from the rediss:// scheme (the response cache also accepts tls: true, which rewrites the scheme). Credentials belong in the URL userinfo and should arrive through the environment, never in a checked-in file. config validate warns about a password written into the config and warns again when credentials travel over a plaintext redis:// endpoint; router logs always mask the password.

Redis TLS certificates¶

The router always verifies the server certificate of a rediss:// endpoint. Verification cannot be disabled, and config loading rejects a URL with an #insecure fragment. The certificate must match the host in the URL, so connect by a name or IP address listed in it.

A Redis endpoint behind a public CA works with no setup: the official image carries a CA bundle at /etc/ssl/certs/ca-certificates.crt and sets SSL_CERT_FILE to it.

For a private CA or a self-signed certificate, mount the PEM file in its own directory and set SSL_CERT_DIR to that directory. Set SSL_CERT_FILE too, so the system bundle stays loaded:

docker run --rm \
  -p 8080:8080 \
  -v "$PWD/config.yaml:/etc/continuum-router/config.yaml:ro" \
  -v "$PWD/redis-ca.pem:/etc/private-ca/redis-ca.pem:ro" \
  -e SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt \
  -e SSL_CERT_DIR=/etc/private-ca \
  -e REDIS_URL \
  ghcr.io/lablup/continuum-router:2.0.0

On Kubernetes, store the file in a Secret and mount it outside the read-only /etc/continuum-router config mount:

kubectl -n continuum-router create secret generic redis-ca --from-file=redis-ca.pem

With the Helm chart:

extraEnv:
  - name: SSL_CERT_FILE
    value: /etc/ssl/certs/ca-certificates.crt
  - name: SSL_CERT_DIR
    value: /etc/private-ca
extraVolumes:
  - name: redis-ca
    secret:
      secretName: redis-ca
extraVolumeMounts:
  - name: redis-ca
    mountPath: /etc/private-ca
    readOnly: true

With a Kustomize overlay:

patches:
  - target:
      kind: Deployment
      name: continuum-router
    patch: |-
      - op: add
        path: /spec/template/spec/volumes/-
        value: {name: redis-ca, secret: {secretName: redis-ca}}
      - op: add
        path: /spec/template/spec/containers/0/volumeMounts/-
        value: {name: redis-ca, mountPath: /etc/private-ca, readOnly: true}
      - op: add
        path: /spec/template/spec/containers/0/env/-
        value: {name: SSL_CERT_FILE, value: /etc/ssl/certs/ca-certificates.crt}
      - op: add
        path: /spec/template/spec/containers/0/env/-
        value: {name: SSL_CERT_DIR, value: /etc/private-ca}

Either variable replaces the system trust store instead of adding to it. Only the files they name are loaded, and files dropped into /etc/ssl/certs are ignored while SSL_CERT_FILE is set. Pointing SSL_CERT_FILE at only the private CA has these effects:

• Redis behind a public CA: Fails verification.
• HTTPS backends and the Continuum Hub connection on Linux: Fail verification, because their reqwest and WebSocket clients read the same variables.

An unreadable path has a harsher result:

• SSL_CERT_FILE unreadable or empty as a value, on Linux: The router exits with a panic at startup (Failed to create HTTP client), because no trust roots load.
• SSL_CERT_DIR unreadable, with a good SSL_CERT_FILE: Every Redis connection fails with an error naming the path, while HTTP clients log a warning and keep running.

On macOS and Windows, reqwest and the Hub WebSocket client use the operating system verifier and ignore these variables, while Redis still reads them. Other clients compiled into some builds (the AWS SDK, GCP authentication, and the Kubernetes client) also depend on the same certificate loader, so treat the variables as process-wide.

When verification fails, the log record's error field says so and names SSL_CERT_FILE. For a host mismatch it also lists the expected host and the names in the certificate, never the password. The router then treats the certificate failure like any other connection failure:

Hub tier minute limits¶

A router connected to a Continuum Hub enforces each key's tier rpm, input_tpm, and output_tpm on its own by default. N routers behind one load balancer therefore admit up to N times a key's limit, and because token limits are charged after the response, a burst of concurrent requests can pass an empty token window. control_plane.policy.minute_limits addresses both, and each part is opt-in:

control_plane:
  policy:
    enabled: true
    minute_limits:
      redis:
        url: "${MINUTE_LIMITS_REDIS_URL}"
        key_prefix: "continuum:minute:"
      reserve_tokens: true

• redis: Shares the three minute windows through one Redis or Valkey endpoint under the contract above. One script checks every limit and charges rpm atomically, and the windows open and expire on the store's clock. All keys of one Hub key share a hash tag, so a cluster behind a proxy keeps them in one slot. The block has its own connection pool and does not turn on rate_limiting.
• reserve_tokens: Reserves each metered request's estimated tokens until its response finishes, and admits a request only while window usage plus reservations is under the limit. With redis, every router sees the other routers' reservations, and a crashed router's reservations expire within 30 seconds.

The token estimate over-reserves on purpose. The prompt is a quarter of the request body size (at most 32,768 tokens), and the output is the request's max_tokens (4,096 when absent, at most 131,072), so a small output_tpm also limits how many requests run at once.

Point redis at a server whose maxmemory-policy is noeviction, or at a server separate from an evicting response cache: an evicted key resets its window without any error. Like the rest of control_plane, a change needs a restart. config validate warns when redis is set in a binary built without redis-cache.

Deployment wiring¶

Neither the Helm chart nor the Kustomize bundle ships a Redis server, a StatefulSet, or a generated credential. They reference a Secret you create and open the NetworkPolicy egress needed to reach the endpoint.

kubectl -n continuum-router create secret generic continuum-router-redis \
  --from-literal=REDIS_URL="rediss://:$REDIS_PASSWORD@cache.example.com:6380"

# Helm: layer the overlay on an environment profile.
helm upgrade --install router deploy/helm/continuum-router \
  --namespace continuum-router \
  -f deploy/helm/continuum-router/values-production.yaml \
  -f deploy/helm/continuum-router/values-shared-redis.yaml \
  --set-string image.digest="$IMAGE_DIGEST"

# Kustomize: apply the overlay instead of the base.
kubectl apply -k deploy/kubernetes/overlays/shared-redis

Both overlays open egress to port 6380 by port number only, because a managed endpoint may live outside the cluster and cannot be selected by namespace or pod label. Narrow that to an ipBlock once you know the address. redis.enabled: true without a Secret name fails the Helm render rather than inventing a credential.

Failure semantics¶

Only fail-closed rate limiting refuses traffic when Redis disappears. Size rate limits against the degraded ceiling: if you run six replicas and the fleet must never exceed 600 rps, an outage with a 600 rps shared limit admits 3600 rps. Reconnection is automatic in every consumer, and no restart is needed after the endpoint returns.

Recovering from a shared-state deployment back to per-replica state is a config change: set storage: memory and backend: memory, reapply, and roll. Counters restart from empty.

Performance profile¶

A safe tuning workflow is:

1. Start from a generated template.
2. Measure direct-provider and routed traffic. For the routed side, perf/run.sh gives a reproducible measurement without a provider account (see "Capacity planning: throughput" above).
3. Tune server.connection_pool_size, timeouts, retries, health checks, and selection_strategy.
4. Size response cache and rate limits from observed traffic.
5. Validate every change. Selection strategy, retry policy, and per-request timeout budgets reload live; restart for startup-built fields such as listener/client construction (timeouts.connection and the shared-client ceiling at timeouts.request.streaming.total).

Security¶

• Use API-key blocking mode for public API traffic.
• Configure Admin, Files, WebUI, and Metrics authentication independently.
• Terminate TLS before the router.
• Keep provider keys in environment/secret stores and reference them with ${ENV_VAR}.
• Bind Admin access to a private network or strict proxy policy.
• Do not publish config show --resolved output; it may contain plaintext credentials.
• Mount configuration read-only unless using Admin configuration save/import operations, which require a writable destination.

The public /health, /health/ready, and /version endpoints contain operational information and do not use API-key authentication. Apply network-layer filtering if even that information must be private.

Observability¶

Enable metrics explicitly and protect them:

metrics:
  enabled: true
  port: 9090
  path: /metrics
  auth:
    method: basic
    basic_auth:
      username: metrics
      password: "${METRICS_PASSWORD}"

Prometheus scrapes the dedicated metrics listener on metrics.port (9090), which serves only the metrics path. /metrics also stays on the API port for existing scrape configurations.

Use logging.format: json for structured production logs. CONTINUUM_LOG_LEVEL and RUST_LOG can adjust verbosity; avoid debug logging of sensitive workloads for long periods.

Admin diagnostics include:

curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:8080/admin/health
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:8080/admin/backends
curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:8080/admin/circuit/all

Backup and recovery¶

Back up the operator-owned state that your configuration actually enables:

• config.yaml or TOML source, kept in version control without secrets
• External API-key file, if configured
• Files API storage and metadata database
• Stats/metrics persistence files
• OAuth token stores
• Control-plane state and its probe-budget sidecar when the feature is active
• Response-cache backing stores when persistence matters

Test restore procedures with the same binary version and feature set. Validate restored configuration before starting.

Troubleshooting¶

Getting a shell¶

The image ships no shell, so docker exec ... sh and kubectl exec ... -- sh do not work. Attach a shell in a separate container that shares the router's namespaces instead. The router's own filesystem is reachable at /proc/1/root.

# Kubernetes: an ephemeral debug container in the running pod
kubectl debug -it <pod> --image=busybox --target=continuum-router

# Docker: a throwaway container sharing the router's PID and network namespaces
docker run --rm -it \
  --pid=container:continuum-router \
  --network=container:continuum-router \
  busybox sh

Most inspection needs no shell at all, because the router's own subcommands run as the image entrypoint:

docker run --rm -v "$PWD/config.yaml:/c.yaml:ro" \
  ghcr.io/lablup/continuum-router:2.0.0 config validate /c.yaml
docker run --rm -v "$PWD/config.yaml:/c.yaml:ro" \
  ghcr.io/lablup/continuum-router:2.0.0 config show --resolved /c.yaml

Container is unhealthy¶

docker exec continuum-router /usr/local/bin/continuum-router --health-check
docker logs continuum-router

Confirm the router is listening on 0.0.0.0:8080 inside a container rather than loopback only.

Configuration fails¶

continuum-router config validate /etc/continuum-router/config.yaml
continuum-router config show /etc/continuum-router/config.yaml

Use config show --resolved only in a protected terminal because it can reveal resolved secrets.

No backend is selected¶

Check the requested model, backend model lists, health, circuit state, and caller allow-lists. Selection strategies only choose among eligible candidates.

See also¶

• Installation
• Configuration
• Performance
• Security
• Metrics