VAST Data Integration Guide¶

Continuum Router integrates with VAST Data storage at two points: the response cache L2 tier, and the KV cache index, which tracks KV tensors that a serving engine has offloaded to VAST. This guide covers both integration points with deployment examples, explains how VAST fits alongside disaggregated prefill/decode serving, and provides a full production configuration.

VAST is not used for prefill-to-decode KV transfer in disaggregated serving. The serving engine transfers KV blocks directly between prefill and decode workers (for example over NIXL in a Dynamo deployment), while the router selects the participating workers; see KV Transfer. The legacy external_storage fields are covered in Legacy external_storage fields.

Table of Contents¶

• VAST Data Connection Methods
• Prerequisites
• Example: Response Cache on VAST (S3 API)
• Example: KV Tensor Offloading (vLLM + VAST)
• Disaggregated Prefill/Decode Alongside VAST
• Example: Full Production Setup
• Performance Benchmarks

VAST Data Connection Methods¶

VAST Data exposes multiple storage access protocols. Continuum Router connects directly only to the S3-compatible API for L2 response caching. KV cache integration runs through the serving engine: the engine offloads and reloads its own KV tensors over its configured storage connector, and the router tracks those tensors through KV cache events rather than reading VAST directly.

For router-level integration, you need an S3-compatible endpoint with a pre-created bucket and credentials for the response cache. For KV offloading, configure a VAST endpoint reachable from the engine hosts on the serving engine itself rather than on the router.

Prerequisites¶

VAST Data Cluster Requirements¶

• VAST Data software version 4.0 or later
• S3 Gateway enabled for response cache integration
• Sufficient capacity for your workload:
  • Response cache: estimate 2–20 KB per unique request; plan for millions of entries
  • KV tensors: estimate 2 × num_layers × num_heads × head_dim × seq_len × 2 bytes per cached prefix

Credentials and Access¶

• S3 access key and secret key with read/write permissions on the target bucket
• Bucket created in advance (the router does not create buckets automatically)
• Network path from each router instance to the VAST cluster

Network Requirements¶

• L3 connectivity between router hosts and the VAST S3 Gateway
• Firewall rules allowing TCP port 443 or 80 from router hosts to the VAST S3 Gateway (response cache)
• For KV offloading: a network path from each engine host to the VAST endpoint used by its KV connector (jumbo frames recommended for large tensor transfers); the router does not need a route to this endpoint
• For NVMe-oF/RDMA: dedicated RDMA NIC and fabric on the engine hosts (not required for router-level integration)

Environment Variables¶

Store sensitive credentials in environment variables rather than configuration files:

export VAST_ACCESS_KEY="your-access-key"
export VAST_SECRET_KEY="your-secret-key"

The response-cache S3 access_key and secret_key fields expand ${ENV_VAR} references when the cache is initialized.

Example: Response Cache on VAST (S3 API)¶

This configuration uses Redis as the hot L1 cache and VAST Data S3 as the durable L2 cache. Responses that miss L1 are looked up in VAST before calling a backend, and L2 hits are promoted back to L1 for faster subsequent access.

How it works¶

1. Client sends a deterministic request (temperature = 0)
2. Router computes a cache key from the request parameters
3. L1 (Redis) is checked; on a hit, the response is returned immediately
4. On an L1 miss, L2 (VAST S3) is checked; on a hit, the response is returned and promoted to L1
5. If both tiers miss, the router calls the backend and stores the response in both L1 and L2

Configuration¶

response_cache:
  enabled: true

  # "tiered" enables the L1 + L2 architecture
  backend: tiered

  # Global TTL for all cache entries
  ttl: "24h"

  # Maximum response body size eligible for caching
  max_response_size: 1048576         # 1 MiB
  max_stream_buffer_size: 10485760   # 10 MiB

  # L1: Redis (hot cache - fast lookup, limited capacity)
  l1:
    type: memory   # or "redis" for distributed L1
    max_value_size: 1048576   # values larger than 1 MiB go directly to L2

  # Redis config used by L1 when l1.type is "redis"
  redis:
    url: "redis://redis-service:6379"
    pool_size: 16
    key_prefix: "cr:resp:"
    fallback_to_memory: true

  # L2: VAST Data S3 (warm cache - high capacity, durable)
  l2:
    type: s3
    endpoint: "https://vast-s3.example.com"
    bucket: "llm-response-cache"
    key_prefix: "response-cache/"
    region: "us-east-1"
    access_key: "${VAST_ACCESS_KEY}"
    secret_key: "${VAST_SECRET_KEY}"
    # Optional: override TTL for L2 entries (default: inherits global ttl)
    ttl_override: "168h"

  # Tiered cache promotion behavior
  tiered:
    promote_on_hit: true       # Promote L2 hits back to L1
    l1_promotion_ttl: "30m"    # TTL for promoted L1 entries

Field Reference¶

Example: KV Tensor Offloading (vLLM + VAST)¶

This configuration enables the KV cache index with storage offloading awareness. When vLLM offloads GPU KV tensors to VAST, the router tracks those entries in the StorageWarm tier and applies a discounted scoring weight compared to GPU-resident (GpuHot) tensors.

How it works¶

1. vLLM computes KV tensors for a prompt and emits a cache_created event (GpuHot tier)
2. Under GPU memory pressure, vLLM offloads tensors to VAST and emits cache_offloaded
3. The KV index downgrades the entry to StorageWarm
4. The router routes subsequent requests with matching prefixes to the backend holding StorageWarm data, applying storage_tier_weight relative to GpuHot
5. vLLM reloads the tensors from VAST and emits cache_reloaded, upgrading the index entry back to GpuHot

Continuum Router Configuration¶

kv_cache_index:
  enabled: true
  backend: memory   # or "redis" for multi-instance deployments

  # Scale max_entries with the number of unique prompts × number of backends
  max_entries: 500000

  # Entries expire after 15 minutes; adjust based on vLLM eviction rate
  entry_ttl_seconds: 900

  scoring:
    overlap_weight: 0.6
    load_weight: 0.3
    health_weight: 0.1

    # Only activate KV-aware routing when a backend holds ≥30% coverage
    min_overlap_threshold: 0.30

    # GPU-resident data gets full overlap credit
    gpu_tier_weight: 1.0

    # Storage-offloaded data is valuable but incurs reload latency - discount it
    storage_tier_weight: 0.6

  # Track GPU hot vs. VAST warm tiers
  storage_offloading:
    enabled: true
    # When vLLM emits only cache_created/cache_evicted (no cache_offloaded),
    # treat evictions as offloads to VAST rather than permanent removal
    treat_eviction_as_offload: true

  # Subscribe to continuum-kv-listener SSE streams for each backend
  event_sources:
    - backend_name: vllm-1
      endpoint: "http://kv-listener.internal:7817/events/vllm-1"
      reconnect_interval_ms: 5000

    - backend_name: vllm-2
      endpoint: "http://kv-listener.internal:7817/events/vllm-2"
      reconnect_interval_ms: 5000

vLLM Launch Command¶

Configure vLLM to emit KV events to continuum-kv-listener and offload KV tensors to VAST Data:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --enable-prefix-caching \
  --kv-cache-dtype auto \
  --max-model-len 32768 \
  --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'

Key flags:

• --enable-prefix-caching: activates KV prefix caching on the backend.
• --kv-cache-dtype auto: uses the model's native dtype for KV cache storage.
• --kv-transfer-config: selects the KV connector used for storage offloading. Upstream vLLM does not include a built-in VastKVConnector (in vllm/distributed/kv_transfer/kv_connector/factory.py as of commit 8c1557a7), so this example uses LMCacheConnectorV1 backed by a VAST storage mount configured in LMCache's own configuration file. Refer to your connector and VAST documentation for exact connector settings.

You must also configure kv_events_config (or --kv-events-config) on vLLM so the engine publishes ZMQ msgpack KVEventBatch frames to the matching continuum-kv-listener source (for example, tcp://kv-listener.internal:5557).

The router subscribes to the listener's HTTP SSE URL rather than directly to vLLM, because vLLM does not expose an HTTP /v1/kv_events endpoint. The listener reconstructs the token prefix and translates vLLM block hashes into the router's SHA-256 prefix key via extract_prefix_key before forwarding events to the router.

Scoring Behavior with VAST Offloading¶

Disaggregated Prefill/Decode Alongside VAST¶

Disaggregated serving separates prefill (prompt processing) from decode (token generation). The serving engine transfers KV blocks computed during prefill directly from the prefill worker to the decode worker over NIXL (in Dynamo deployments), NCCL, or its storage tier. The router never reads, writes, or addresses tensor bytes; it selects the prefill worker, decode worker, and routing path as described in Disaggregated Prefill/Decode Serving.

VAST does not sit in the prefill-to-decode transfer path. Instead, it serves the same role in a disaggregated deployment as in a unified deployment: an offload storage tier tracked by the KV cache index as StorageWarm (see KV Tensor Offloading). When a decode worker has offloaded a prefix to VAST, the fast_decode routing path can route matching requests to a decode worker whose engine reloads the offloaded prefix instead of running prefill again.

How it works¶

1. The router checks the KV index for the request's prefix key (requires prefix_routing.enabled: true)
2. If a decode worker holds GPU-resident tensors (GpuHot), the router routes directly to that worker (fast_decode)
3. If the engine has offloaded the tensors to VAST (StorageWarm), the router routes to the least-loaded healthy decode worker so its engine reloads them from storage (fast_decode)
4. If no cached tensors exist, the router selects a prefill worker and a decode worker (prefill_then_decode), leaving the KV transfer to the engine

The generic prefill-then-decode wire protocol is not yet implemented in this release: without an engine-specific picker, prefill_then_decode selects both workers and then returns 503 disaggregated_prefill_decode_unavailable (or falls back to unified serving when fallback_to_unified is enabled).

For an end-to-end disaggregated deployment today, run a Dynamo fleet behind a direct-mode shim. In that setup, the router selects the prefill and decode worker pair while the Dynamo frontend orchestrates the prefill hop, the NIXL transfer, and the decode hop; see Routing through a direct-mode shim.

Configuration¶

prefix_routing:
  enabled: true

kv_cache_index:
  enabled: true
  storage_offloading:
    enabled: true   # track engine-offloaded tensors (e.g. on VAST) as StorageWarm

disaggregated_serving:
  enabled: true

  # Timeout for the prefill computation phase
  prefill_timeout: "60s"

  # Timeout for the engine-owned KV transfer between workers
  kv_transfer_timeout: "15s"

  # Fall back to unified backends if disaggregated backends are unavailable
  fallback_to_unified: true

backends:
  # Prefill workers: compute KV tensors for input prompts
  - name: prefill-worker-1
    url: "http://vllm-prefill-1:8000"
    role: prefill

  - name: prefill-worker-2
    url: "http://vllm-prefill-2:8000"
    role: prefill

  # Decode workers: generate tokens from the KV blocks the engine transferred
  - name: decode-worker-1
    url: "http://vllm-decode-1:8000"
    role: decode
    weight: 2

  - name: decode-worker-2
    url: "http://vllm-decode-2:8000"
    role: decode
    weight: 2

  - name: decode-worker-3
    url: "http://vllm-decode-3:8000"
    role: decode
    weight: 2

  # Unified fallback: handles both phases when disaggregated pool is unavailable
  - name: unified-fallback
    url: "http://vllm-unified:8000"
    role: unified

No VAST endpoint is needed in this router configuration. Each worker's KV offloading to VAST is configured on the engine as shown in KV Tensor Offloading, and its KV events reach the router through kv_cache_index.event_sources.

Worker Configuration¶

Prefill-to-decode KV transfer is configured on the serving engine rather than on the router. In a Dynamo deployment, workers run in prefill and decode disaggregation modes and exchange KV blocks over NIXL; deploy/examples/dynamo-direct-shim/ provides a runnable, GPU-free example of this topology. If an engine also offloads KV tensors to VAST, configure that storage connector alongside its transfer settings on the engine.

Routing Path Headers¶

Pool Sizing Recommendations¶

• Start with a 3:1 decode-to-prefill ratio: decode workers process tokens longer than prefill workers process prompts
• Increase the prefill pool for workloads with high prompt diversity (many unique system prompts)
• Increase the decode pool for workloads with long output sequences or high concurrency

Legacy external_storage fields¶

VAST does not act as a router-addressed transit layer where prefill workers write tensors to {endpoint}/{kv_namespace}/{prefix_hash} for decode workers to read. No supported engine consumes that reference format.

The disaggregated_serving.default_external_storage and per-backend external_storage fields are retained solely for configuration compatibility and are ignored on the request path:

disaggregated_serving:
  enabled: true
  # Legacy: accepted for compatibility, not read by the request path
  default_external_storage:
    endpoint: "http://vast-cluster:8080"
    kv_namespace: "inference/kv-cache"
    # credentials: "${VAST_CREDENTIALS}"   # a single string, not a map

backends:
  - name: prefill-worker-1
    url: "http://vllm-prefill-1:8000"
    role: prefill
    # Legacy: accepted for compatibility, not read by the request path
    external_storage:
      endpoint: "http://vast-cluster:8080"
      kv_namespace: "inference/kv-cache"

Configurations containing these fields continue to load without error, and removing them has no runtime effect. See external_storage (per-backend, legacy) for field details.

Example: Full Production Setup¶

This configuration combines both VAST integration points (tiered response cache with Redis L1 + VAST S3 L2, and the KV cache index with storage offloading awareness) alongside disaggregated prefill/decode serving. The prefill-to-decode KV transfer remains engine-owned.

server:
  bind_address: "0.0.0.0:8080"

selection_strategy: LeastLatency

# ─── Prefix-Aware Routing (Tier 1) ───────────────────────────────────────────
prefix_routing:
  enabled: true
  max_prefix_length: 2048
  load_factor_epsilon: 0.20

# ─── Tiered Response Cache (Tier 2 + VAST S3 L2) ─────────────────────────────
response_cache:
  enabled: true
  backend: tiered
  ttl: "24h"
  max_response_size: 2097152       # 2 MiB
  max_stream_buffer_size: 20971520 # 20 MiB

  # L1: Redis (hot, fast, limited capacity)
  l1:
    type: redis
    max_value_size: 524288   # Values >512 KiB go directly to L2

  redis:
    url: "redis://redis-cluster:6379"
    pool_size: 32
    key_prefix: "cr:resp:"
    connect_timeout_ms: 3000
    command_timeout_ms: 1000
    fallback_to_memory: true

  # L2: VAST Data S3 (warm, durable, high capacity)
  l2:
    type: s3
    endpoint: "https://vast-s3.prod.example.com"
    bucket: "prod-llm-response-cache"
    key_prefix: "response-cache/"
    region: "us-east-1"
    access_key: "${VAST_ACCESS_KEY}"
    secret_key: "${VAST_SECRET_KEY}"
    ttl_override: "168h"

  tiered:
    promote_on_hit: true
    l1_promotion_ttl: "30m"

# ─── KV Cache Index with VAST Offloading Awareness (Tier 4) ──────────────────
kv_cache_index:
  enabled: true
  backend: redis   # Reuses the redis pool from response_cache.redis
  max_entries: 1000000
  entry_ttl_seconds: 1800

  scoring:
    overlap_weight: 0.6
    load_weight: 0.3
    health_weight: 0.1
    min_overlap_threshold: 0.25
    gpu_tier_weight: 1.0
    storage_tier_weight: 0.6

  storage_offloading:
    enabled: true
    treat_eviction_as_offload: true

  event_sources:
    - backend_name: prefill-worker-1
      endpoint: "http://kv-listener.internal:7817/events/prefill-worker-1"
      reconnect_interval_ms: 5000
    - backend_name: prefill-worker-2
      endpoint: "http://kv-listener.internal:7817/events/prefill-worker-2"
      reconnect_interval_ms: 5000
    - backend_name: decode-worker-1
      endpoint: "http://kv-listener.internal:7817/events/decode-worker-1"
      reconnect_interval_ms: 5000
    - backend_name: decode-worker-2
      endpoint: "http://kv-listener.internal:7817/events/decode-worker-2"
      reconnect_interval_ms: 5000
    - backend_name: decode-worker-3
      endpoint: "http://kv-listener.internal:7817/events/decode-worker-3"
      reconnect_interval_ms: 5000

# ─── Disaggregated Prefill/Decode Serving ────────────────────────────────────
disaggregated_serving:
  enabled: true
  prefill_timeout: "60s"
  kv_transfer_timeout: "15s"   # engine-owned transfer between workers
  fallback_to_unified: true

# ─── Backends ─────────────────────────────────────────────────────────────────
backends:
  - name: prefill-worker-1
    url: "http://vllm-prefill-1:8000"
    role: prefill
    models: ["meta-llama/Llama-3.1-70B-Instruct"]

  - name: prefill-worker-2
    url: "http://vllm-prefill-2:8000"
    role: prefill
    models: ["meta-llama/Llama-3.1-70B-Instruct"]

  - name: decode-worker-1
    url: "http://vllm-decode-1:8000"
    role: decode
    weight: 2
    models: ["meta-llama/Llama-3.1-70B-Instruct"]

  - name: decode-worker-2
    url: "http://vllm-decode-2:8000"
    role: decode
    weight: 2
    models: ["meta-llama/Llama-3.1-70B-Instruct"]

  - name: decode-worker-3
    url: "http://vllm-decode-3:8000"
    role: decode
    weight: 2
    models: ["meta-llama/Llama-3.1-70B-Instruct"]

  - name: unified-fallback
    url: "http://vllm-unified:8000"
    role: unified
    models: ["meta-llama/Llama-3.1-70B-Instruct"]

# ─── Health Checks ────────────────────────────────────────────────────────────
health_checks:
  interval: "15s"
  timeout: "5s"
  unhealthy_threshold: 3
  healthy_threshold: 2
  endpoint: "/health"

logging:
  level: info
  format: json

Worker Launch Notes for Full Production Setup¶

• KV offloading to VAST: Configure each prefill and decode worker that offloads KV tensors to VAST using a KV connector as shown in KV Tensor Offloading, pointed at the production VAST endpoint and publishing KV events to the listener listed in kv_cache_index.event_sources.
• Prefill-to-decode transfer: Configure inter-worker transfer on the engine (such as NIXL between workers in a Dynamo deployment); see Worker Configuration. VAST is not part of this transfer path.
• Unified fallback: Run a standard serving engine with prefix caching enabled:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 4 \
  --enable-prefix-caching \
  --kv-cache-dtype auto \
  --max-model-len 32768

Because the unified fallback handles both prefill and decode locally, it does not participate in inter-worker KV transfer.

Performance Benchmarks¶

The following figures are illustrative estimates based on typical LLM serving workloads. Actual results depend on prompt diversity, GPU memory capacity, network bandwidth to VAST, and concurrent request volume.

Response Cache Hit Rate (with VAST L2)¶

L2 benefits workloads where identical requests recur over hours or days beyond L1 TTL.

Disaggregated Serving Latency¶

Latency improvements compared to unified (single-worker) inference on a 70B parameter model:

The prefill_then_decode path adds overhead on the first request for a prefix, but subsequent requests with the same prefix benefit from the fast_decode path.

KV Cache Index Routing Effectiveness¶

VAST Data Access Latency¶

For production deployments with strict latency requirements on the offload and reload path, place the VAST cluster in the same rack or use RDMA-capable network adapters with NVMe-oF accessed directly from the engine hosts. Prefill-to-decode transfer does not touch VAST; its latency is governed by the inter-worker fabric (NIXL over RDMA or NCCL).