Disaggregated Prefill/Decode Serving¶

Continuum Router supports disaggregated inference, where the prefill phase (prompt processing) and the decode phase (token generation) run on separate GPU workers. The serving engine transfers KV blocks computed during prefill directly between workers over its own transport, such as NIXL in a Dynamo deployment. The router never touches tensor bytes; it selects the prefill worker, the decode worker, and the routing path, then reports the decision in the response headers.

Overview¶

In standard (unified) inference, a single GPU worker handles both phases of a request:

1. Prefill: compute key-value attention tensors for the full input prompt
2. Decode: autoregressively generate output tokens, reading the KV cache computed in step 1

Disaggregated serving separates these phases across specialized workers:

• Prefill workers: high-throughput GPUs optimized for batch KV computation
• Decode workers: GPUs holding warm KV caches optimized for low-latency token generation
• Engine-owned KV transfer: the serving engine moves KV blocks from the prefill worker to the decode worker over NIXL, NCCL, or its storage tier, while the router selects which two workers participate

This separation is particularly effective when:

• Many requests share the same long system prompt (e.g., RAG documents, tool definitions)
• Prefill and decode workloads have different GPU memory requirements
• Token generation latency is the primary optimization target

Request Flow¶

The DisaggregatedOrchestrator selects a routing path once per incoming chat completion request, before the streaming (stream: true) and buffered pipelines diverge. It passes the chosen backend to the standard dispatch pipeline as a single-element allow-list, so every downstream check (health, circuit breaker, per-backend rate limit, and Hub credential binding) still applies.

The orchestrator queries the KV index using the 64-character hex SHA-256 prefix key, matching the key published by the KV listener. Because that key is computed only when prefix_routing.enabled is true, fast-decode routing requires prefix routing to be enabled. When prefix routing is off, the orchestrator skips the KV index lookup and proceeds directly to prefill-then-decode selection.

Routing Paths¶

The selected routing path is reported in the X-Continuum-Routing-Path response header. The generic prefill-then-decode wire protocol is not yet implemented in this release: when an engine-specific picker is not active, that path selects both workers and then returns 503 disaggregated_prefill_decode_unavailable or falls back to unified serving.

The unified path value is written only by Dynamo shim dispatch when no prefill worker serves the model and the router pins a decode or aggregated worker instead. This path requires a binary built with the dynamo feature and a configured dynamo.shim.backends pool.

When no backend is configured with a prefill or decode role, the orchestrator bypasses disaggregated routing before writing any headers or metrics, and the request proceeds through standard unified serving.

Response Headers¶

Backend name headers follow the router-wide X-Served-Backend disclosure policy: they are included only when the client sends an X-Backend request header, preventing unintended topology disclosure. Sending an empty X-Backend header opts in to backend name headers without expressing a routing preference.

This disclosure policy applies equally to buffered and streaming responses. X-Continuum-Routing-Path identifies a routing path rather than a backend and is always written when disaggregated routing runs.

Backend Roles¶

Each backend in the configuration can be assigned a role:

When disaggregated serving is enabled, the backend pool registers RoleFilterScorer to enforce role assignment. This scorer assigns f64::NEG_INFINITY to backends that are incompatible with the active inference phase, while keeping unified backends eligible for all phases.

Because RoleFilterScorer reads roles from the live pool configuration, hot-reloading a backend's role (with the same name and URL) takes effect on the next request without draining or rebuilding the backend.

Hot-reloading models, weight, or request_extensions drains and rebuilds the pooled backend while preserving its engine-statistics snapshot. Changing a backend's URL resets both the engine-statistics snapshot and any control-plane capacity state.

KV Transfer¶

The serving engine manages KV transfer between the prefill worker and the decode worker. Dynamo deployments transfer KV blocks over NIXL, while other engines use NCCL or their own storage tier. The router never reads, writes, or addresses tensor data.

During a PrefillThenDecode flow, the router:

1. Selects the least-loaded healthy prefill worker and the least-loaded healthy decode worker for the request
2. Reports both workers in X-Continuum-Prefill-Backend and X-Continuum-Decode-Backend when the request includes X-Backend
3. Delegates the KV transfer itself to the engine

Legacy external storage fields¶

The disaggregated_serving.default_external_storage and per-backend external_storage fields, along with the KvReference type ({endpoint}/{kv_namespace}/{prefix_hash}, storage path, prefix hash, token count, and tensor format), are retained for configuration compatibility and documented under Configuration Reference. No supported engine consumes that reference, and the request path ignores these fields.

Configuration¶

Top-Level Disaggregated Serving¶

disaggregated_serving:
  enabled: false                # Enable disaggregated prefill/decode serving
  prefill_timeout: "30s"        # Timeout for the prefill phase
  kv_transfer_timeout: "10s"    # Reserved; the engine owns the KV transfer
  fallback_to_unified: true     # Fall back to unified when disaggregated unavailable

  # Legacy: default external storage for backends that do not specify their
  # own. Accepted for compatibility; the request path does not read it.
  default_external_storage:
    endpoint: "http://storage-cluster:8080"
    kv_namespace: "inference/kv-cache"
    # credentials: "${STORAGE_CREDENTIALS}"   # Optional, a single string

Per-Backend Role Assignment¶

Add role to each backend (external_storage is a legacy field, accepted but not read):

backends:
  # Prefill worker - computes KV tensors
  - name: prefill-worker-1
    url: "http://vllm-prefill-1:8000"
    role: prefill

  # Decode workers - generate tokens using cached KV data
  - name: decode-worker-1
    url: "http://vllm-decode-1:8000"
    role: decode
    weight: 2

  - name: decode-worker-2
    url: "http://vllm-decode-2:8000"
    role: decode
    weight: 2

  # Unified fallback backend (optional)
  - name: unified-fallback
    url: "http://vllm-unified:8000"
    role: unified   # or omit - unified is the default

Minimal Example¶

Prefix routing must be enabled for fast-decode routing to query the KV index; without it, every request uses prefill-then-decode selection.

prefix_routing:
  enabled: true

kv_cache_index:
  enabled: true

disaggregated_serving:
  enabled: true

backends:
  - name: prefill-gpu
    url: "http://vllm-prefill:8000"
    role: prefill
  - name: decode-gpu-1
    url: "http://vllm-decode-1:8000"
    role: decode
  - name: decode-gpu-2
    url: "http://vllm-decode-2:8000"
    role: decode

Configuration Reference¶

disaggregated_serving¶

external_storage (per-backend, legacy)¶

Accepted for configuration compatibility with the earlier external-storage transfer design. The request path does not read these fields.

role (per-backend)¶

Metrics¶

The global /metrics endpoint exports the disaggregated_ Prometheus metric family. For both buffered and streaming requests, disaggregated_requests_total increments once per request handled on a disaggregated path and does not increment when a request falls back to unified serving.

Label values outside the defined sets are recorded as unknown to bound metric cardinality. Because routing_path="unified" is emitted only by the Dynamo shim's aggregated fallback (see Routing Paths), deployments without a shim pool will never observe that label value.

The duration histograms disaggregated_prefill_duration_seconds{backend}, disaggregated_decode_duration_seconds{backend}, and disaggregated_kv_transfer_duration_seconds are registered for engine-specific pickers and are not populated by the generic orchestrator.

Integration with KV Cache Index¶

Disaggregated serving works alongside the KV Cache Index (Tier 4). The KV index tracks which decode workers hold warm GPU caches for specific prefix keys, allowing the orchestrator to skip the prefill phase when the prefix is already resident in a decode worker's GPU memory (GpuHot tier).

When storage_offloading.enabled is true in the KV cache index configuration, the orchestrator can also route to decode workers holding data in the StorageWarm tier and let the engine reload it on demand.

Backend Selection for Load Balancing¶

Within each phase, the orchestrator selects the least-loaded healthy backend using live pool state:

• Candidates: user-routable backends (excluding internal: true and enabled: false), filtered by the API key's allow-list and then by RoleFilterScorer for the active phase.
• Health: the health checker's current status per backend. Unhealthy workers are excluded; when health checks are disabled, all backends are treated as healthy.
• Load: the pool's active in-flight request gauge, shared with the Admin API and the CHWBL load cap. Ties are broken by configuration order.
• Engine statistics: when the engine_stats poller has a fresh snapshot, waiting_requests and kv_cache_usage are attached to each candidate for engine-aware pickers, while the default least-loaded selection uses the in-flight request count.

This keeps GPU utilization balanced independently across the prefill and decode worker pools.

Fallback Behavior¶

When fallback_to_unified: true (the default), the router falls back to unified serving when no disaggregated path is available or when the routed dispatch fails:

1. Unified role filtering: Fallback requests are restricted to user-routable unified backends that the caller is permitted to reach. Prefill-only and decode-only workers are excluded from the fallback pass, and any x-backend header naming a role-specific worker is ignored.
2. Failed backend exclusion: The backend whose routed dispatch just failed is excluded from the fallback retry while other candidates remain. Dynamo shim pools are an exception: because a shim backend represents a pool of worker replicas rather than a single worker, it remains eligible.
3. Readiness check: The unified-only restriction applies only when at least one unified backend serves the model and passes current health, circuit-breaker, per-backend rate-limit, and Hub credential checks. Otherwise, the router falls back to the standard backend pool without role filtering (still excluding the failed backend unless it is the sole candidate).
4. Scope and allow-list intersection: Fallback filtering narrows the candidate set by intersecting with the API key's allowed_backends, Hub routing decisions, and model-experiment pins; it never widens permissions. Model fallback hops (fallback.fallback_chains) evaluate the caller's original allow-list, so chain targets served only by role workers remain reachable.
5. Observability: Fallback responses report fallback in X-Continuum-Routing-Path and increment disaggregated_fallback_total.

The readiness check in step 3 is a point-in-time snapshot taken before unified dispatch and does not evaluate the engine admission gate (routing.engine_load.admission). If a unified backend passes the snapshot check but fails or is rejected during dispatch, the request returns that selection or 5xx error (or advances a configured fallback chain) without retrying role-specific workers.

Behavior when fallback_to_unified: false¶

When fallback_to_unified is false, failed requests return a 503 error envelope (disaggregated_unavailable, disaggregated_prefill_decode_unavailable, disaggregated_backend_mismatch, or disaggregated_decode_failed) with X-Continuum-Routing-Path: fallback. If a selected decode worker fails on a streaming request before the stream starts, the error response preserves the attempted routing path in X-Continuum-Routing-Path.

Streaming vs. buffered error handling¶

On the streaming path, only transport failures, 5xx responses, 429, and 408 before stream start are treated as decode failures that trigger unified fallback. 429 and 408 reflect transient capacity or rate-limit state on the pinned backend, so another backend can still serve the request.

Any other 4xx response on the streaming path (such as a 400 validation error, 403, or 413) is returned directly to the client under the routed path and is not counted as a decode error. On the buffered path, any failed dispatch, including a 4xx response, is treated as a decode failure.

Guardrails interaction¶

When guardrails are configured, input evaluation runs once in the chat input gate before disaggregated routing begins. If a disaggregated dispatch fails and falls back to unified serving, the retry reuses the original input decision without running a second guardrail input check. On the buffered path, output guardrails evaluate whatever final response is returned to the client, including unified fallback responses.

Deployment Recommendations¶

• GPU allocation: Prefill workers benefit from high memory bandwidth; decode workers benefit from large GPU memory for KV cache residence.
• KV transfer sizing: Estimate the KV size the engine moves per request as 2 * num_layers * num_heads * head_dim * seq_len * 2 bytes (fp16). For a 7B model with 32 layers and 4096-token context: ~256 MB per request, which is what the inter-worker fabric (NIXL over RDMA, NCCL) must sustain.
• Health checks: Configure backend health checks to detect GPU OOM and driver errors quickly so unhealthy phase workers leave the eligible candidate set.
• Decode pool size: More decode workers reduce queuing latency for the token generation phase. A 3:1 decode-to-prefill ratio is a common starting point.