Dynamo Worker Discovery¶

Continuum Router discovers an NVIDIA Dynamo worker fleet by watching the etcd cluster or Kubernetes resources where workers register. It learns which prefill and decode instances exist, which model each serves, and what capacity each advertises, keeping that view current through watch events.

This page covers the following subsystems:

• Worker discovery (etcd, kubernetes, and kubernetes_pods): Watches etcd keys, DynamoWorkerMetadata custom resources joined with EndpointSlices, or operator-labeled Pods.
• Load signals (kv_metrics): Subscribes to per-worker ZMQ publishers (directly or through an event broker) to track active decode blocks and KV occupancy.
• Direct-mode shim routing: Picks a prefill and decode worker per request and dispatches through a dynamo.frontend --router-mode direct shim.
• KV-overlap-aware prefill selection (dynamo-kv): Indexes kv-events block streams and renders prompts with pinned tokenizers to credit prefix cache overlap on prefill workers, or on aggregated workers when the model has no prefill/decode pool.
• In-cluster event broker listener (dynamo-event-bus): Accepts outbound HTTP/2 over TLS tunnels from continuum-event-broker and continuum-kv-listener when the router runs outside the GPU cluster.

The read-only contract¶

The router is a passive observer in Dynamo's discovery protocol. It issues range gets and watches, and nothing else:

• It never takes or renews a lease.
• It never writes a key.
• It never deletes a key.

Worker liveness stays entirely with the workers. Each worker holds its own etcd lease (10 seconds by default), and its keys vanish when that lease expires. The router sees the expiry as an ordinary delete event and removes the instance from its registry. A misbehaving or misconfigured router cannot deregister a healthy worker, extend a dead worker's lifetime, or corrupt the registry read by other Dynamo components.

Everything in src/dynamo/watcher.rs follows this rule: there is no put, delete, lease_grant, or lease_keep_alive anywhere in the module.

Why etcd¶

On bare metal, Dynamo workers register only in etcd, so the router reads etcd to discover the fleet (dynamo.discovery.source: etcd, the default). Single-node setups can also use Dynamo's file store.

On Kubernetes, deployments publish the same information as DynamoWorkerMetadata objects plus EndpointSlices, which the kubernetes source reads (see Kubernetes metadata source). Where the router can read Pods but not those custom resources, the kubernetes_pods source reads operator-labeled Pods instead (see Kubernetes Pod source).

etcd key layout¶

Three prefixes are watched, all parameterized by the configured namespace (Dynamo's DYN_NAMESPACE, default dynamo). The first two describe workers; the third describes their load and KV event publishers:

An Endpoint record looks like this:

{
  "type": "Endpoint",
  "namespace": "dynamo",
  "component": "prefill",
  "endpoint": "generate",
  "instance_id": 7465265302745285,
  "transport": { "tcp": "10.0.0.4:31000/1a7f2b/dynamo/prefill/generate" },
  "device_type": "cuda",
  "request_plane_codec": "msgpack"
}

A Model record carries the card nested as JSON:

{
  "type": "Model",
  "namespace": "dynamo",
  "component": "prefill",
  "endpoint": "generate",
  "instance_id": 7465265302745285,
  "card_json": {
    "display_name": "Qwen/Qwen3-32B",
    "worker_type": "Prefill",
    "kv_cache_block_size": 64,
    "runtime_config": {
      "total_kv_blocks": 12000,
      "max_num_batched_tokens": 8192,
      "max_num_seqs": 256,
      "context_length": 32768,
      "data_parallel_start_rank": 0,
      "data_parallel_size": 1,
      "disaggregated_endpoint": "dynamo/decode/generate"
    }
  }
}

The two records join on (component, endpoint, instance_id):

• A worker's card is the base card registered for the same component and endpoint as its generate record.
• instance_id may be written as a JSON number (the etcd lease ID) or as a hexadecimal string; both normalize to the same key.
• A legacy card that leaves both component and endpoint empty joins on instance_id alone, and a scoped card for the same instance takes precedence.
• A card with a non-empty model_suffix (a LoRA adapter card) never joins.

What becomes a worker¶

Four rules decide whether a record materializes an entry in the registry:

1. Only generate endpoints. A worker registers several endpoints (generate, load_metrics, clear_kv_blocks, and others). Only the generate inference endpoint describes a routable worker; the rest are skipped silently.
2. Component and card must agree. A component named prefill or decode claims that role, and a card's worker_type claims one too. When both are present and disagree, the worker is excluded and the disagreement is logged with both roles. Any other component name lets the card decide on its own, and a worker with no card yet takes the role its component implies.
3. A parse failure skips one record, never the fleet. Non-JSON values or Endpoint records without an instance_id are logged at warn and dropped without affecting other records or stopping the watcher.
4. One identity, one record. Records that claim an identity from which upstream derives a single key are refused together rather than resolved by key order.

Duplicate identities¶

The reducer holds records by etcd key but joins and publishes them by an identity read from the value, so two keys can claim one identity. Legitimate multiplicity was verified against ai-dynamo/dynamo commit ff3ac59e (lib/runtime/src/discovery/kv_store.rs, lib/runtime/src/discovery/mod.rs, lib/runtime/src/storage/kv/etcd.rs, lib/llm/src/local_model.rs, lib/runtime/src/transports/event_plane/mod.rs):

Every record in a refused set is dropped because the router cannot tell a genuine record from a forged one:

• Refused generate instance ID: Produces no worker.
• Refused base cards: Leave the worker materialized as if it had no card (never excluded).
• Refused publisher ID: Gets no subscription, so that worker always reads as quiet under the best-effort contract.

Conflicts increment continuum_dynamo_discovery_events_total{kind="duplicate"} and log a warn message with the record kind, identity, conflict count, and up to four keys. Each distinct conflicting key set is logged once; when the extra records are deleted, the surviving record is used again on the next pass.

Get-then-watch and resync¶

One discovery pass runs four steps:

1. Connect a read-only etcd client.
2. Range-get v1/instances/<ns>/, v1/mdc/<ns>/, and v1/event_channels/namespace/<ns>/, recording the store revision from each response.
3. Open a watch on each prefix starting at revision + 1.
4. Fold every event into the registry until the stream ends.

The revision cursor makes the pass gapless: everything up to the range-get revision is in the snapshot, and everything after it arrives on the watch.

When a pass ends (due to a dropped connection, cancelled watch, or compacted start revision), the router waits for a bounded backoff and starts a new pass from scratch. A full resync discards held records and rebuilds them from fresh range gets, which recovers cleanly even after etcd compaction removes missed events.

• Backoff reset rule: The delay doubles on repeated failures and resets to initial only after all three watches stay open for at least watch_reconnect_backoff.max. This prevents an etcd proxy that answers range gets but immediately closes watch streams from triggering floor-rate reconnect loops (src/dynamo/backoff.rs).
• Timestamp preservation: The reducer is created once per watcher and reused across reconnects so a worker's first_seen timestamp survives resyncs. When the supervisor replaces the watcher after a namespace or connection change, both the reducer and registry reset; a backoff-only change keeps the existing snapshot until the replacement's first pass completes.
• Seed timeout: Each pass bounds its connect-and-seed phase (dial, optional Authenticate RPC, and range gets) to a fixed 30s timeout, while leaving the subsequent long-lived watch streams unbounded.

Wire compatibility (N-2)¶

Dynamo guarantees an N-2 wire window, so the parser tolerates records from workers a couple of releases ahead or behind:

• Every field is optional and uses a safe default when absent.
• Unknown fields are ignored rather than rejected.
• An unrecognized worker_type parses as unknown (no role claim). Beside a prefill or decode component, the component decides; without a component claim, the worker remains unknown (visible to operators, never routed to).
• If the typed subset of a model card fails to deserialize, the worker is still materialized with the raw card retained verbatim.
• The full model card is stored opaquely alongside parsed fields so future consumers can read new fields without a schema migration.

When a card carries a release string (dynamo_version, runtime_version, version, or release), it is logged once per instance at debug.

Bounded memory¶

Every etcd value under the watched prefixes is fleet-authored, so the reducer treats it as untrusted input and enforces three compiled-in constants (src/dynamo/apply.rs):

• MAX_RECORD_BYTES (16 MiB): Raw etcd values above this limit are refused before parsing (an order of magnitude above etcd's default 1.5 MiB --max-request-bytes).
• MAX_RETAINED_CARD_BYTES (1 MiB): Caps the opaque model card retained alongside typed fields. An oversized card drops only the opaque blob while still publishing the worker and its typed subset (worker_type, kv_cache_block_size, runtime_config).
• MAX_RECORDS_PER_KEYSPACE (16384): Caps distinct etcd keys per keyspace (instances and model cards independently). At capacity, updates to existing keys still apply so live workers can heartbeat, while new keys are refused and logged once per keyspace.

The published card_json is wrapped in an Arc<serde_json::Value> once at parse time rather than deep-copied on every materialize() call.

Configuration¶

dynamo:
  enabled: true
  discovery:
    etcd:
      endpoints: ["http://etcd-0.dynamo.svc:2379", "http://etcd-1.dynamo.svc:2379"]
      namespace: "dynamo"          # must match the workers' DYN_NAMESPACE
      # namespaces: ["prod", "canary"]   # or several graphs, each with its own shim pool below
      # tls:
      #   ca_cert: "/etc/etcd/ca.pem"
      #   client_cert: "/etc/etcd/client.pem"
      #   client_key: "/etc/etcd/client-key.pem"
      #   server_name: "etcd.internal"
      # auth:
      #   username: "router"
      #   password: "${ETCD_PASSWORD}"
      watch_reconnect_backoff:
        initial: "500ms"
        max: "30s"
  load:
    max_staleness: "3s"            # how old a worker's last ActiveLoad sample may be and still count as fresh
    direct_allow_cidrs:             # worker Pod networks; omit to preserve the historical allow-any behavior
      - "10.0.0.0/8"
  shim:
    backends: [dynamo-frontend]    # configured OpenAI-wire backends that are `dynamo.frontend --router-mode direct` replicas
    # pools:                       # with several namespaces: one pool per namespace, instead of `backends`
    #   - {namespace: "prod", backends: [dynamo-frontend-prod]}
    #   - {namespace: "canary", backends: [dynamo-frontend-canary]}
    model_map:                     # optional: client model id -> Dynamo display_name, when they differ
      qwen3-32b: "Qwen/Qwen3-32B"

Discovery requires both the dynamo Cargo feature (included in official release binaries) and dynamo.enabled: true. Builds without the feature still parse and validate the section and log a startup warning that discovery is inactive.

Validation rules:

• endpoints: Accepts http(s)://host:port URLs and bare host:port (IPv6 as [::1]:2379), non-empty when enabled, up to 16 entries. Must not embed credentials (user:pass@host or user@host), query strings, fragments, or paths (even empty trailing ? or #). Bare host:port normalizes to http:// before validation.
• Scheme consistency: endpoints cannot mix http:// and https://, and http:// cannot be combined with a tls block.
• namespace / namespaces: Must be non-empty without /, whitespace, or control characters. namespaces accepts up to 16 unique entries (MAX_DYNAMO_NAMESPACES) and is mutually exclusive with a non-default namespace. With multiple namespaces, shim.pools is required instead of shim.backends.
• TLS and auth: client_cert and client_key must be set together. client_key is masked in config readbacks (config show, Admin GET, MCP), while ca_cert and client_cert stay readable. Setting auth over plaintext (http:// without tls) logs a validation warning because etcd's Authenticate RPC and session token travel in cleartext.
• Timings and shim limits: watch_reconnect_backoff.max must be >= initial. load.max_staleness must be between 100ms and 5m. shim.backends accepts up to 64 unique configured backends[].name entries, and shim.model_map accepts up to 1024 non-empty key/value pairs. Listing a shim pool while dynamo.enabled or disaggregated_serving.enabled is false, or referencing an internal: true / enabled: false replica, emits a validation warning.
• Direct dial boundary: load.direct_allow_cidrs accepts up to 64 IPv4 or IPv6 CIDRs. In direct mode the connector resolves each publisher hostname once and dials only an allowed resolved socket address, so DNS cannot change the address between the policy check and the connection. An unset list preserves the previous behavior and makes config validate suggest adding the worker Pod CIDRs. Direct mode assumes in-cluster placement; startup warns when its etcd endpoints appear external. Use broker mode when the router is outside the cluster.

Reload class¶

Nothing in dynamo requires a process restart:

• Immediate reload (no watcher restart): load.max_staleness, shim.backends, shim.pools, and shim.model_map take effect immediately on the next freshness check or request selection without touching the discovery watcher or registry. A load.transport or load.direct_allow_cidrs change restarts only the load and KV subscription tasks under the new dial policy.
• Live watcher replacement: Changes to enabled, discovery.etcd.namespace, connection settings (endpoints, tls, auth), or watch_reconnect_backoff replace the running watcher, log the restart reason (enabled, namespace, connection, backoff, or other) at info, and increment continuum_dynamo_discovery_events_total{kind="restart"}. Disabling discovery logs that discovery stopped without incrementing restart.
• Backoff-only exception: Changing only watch_reconnect_backoff keeps the current registry snapshot readable until the replacement watcher finishes its first full resync.
• Multi-namespace pruning (namespaces): Adding or removing namespaces in discovery.etcd.namespaces or discovery.kubernetes.dynamo_namespaces (when connection settings and kept namespaces' models are unchanged) prunes only removed namespaces' workers and publishers from the snapshot (reason="namespaces"). Kept graphs stay routable without interruption while the replacement watcher resyncs. The shim path also verifies that a worker's namespace and source match the live config before pinning.
• Startup and shutdown: Reloads published during startup are reconciled as soon as the supervisor subscribes. Stopping a watcher waits up to a 5s grace period before aborting, so discovery shutdown takes at most 10s (one in-flight transition plus final stop).

Note that an Admin config import that changes endpoints while leaving auth.password masked reuses the live password on the new host; only point it at a trusted cluster over tls with a pinned ca_cert.

Several Dynamo namespaces¶

A single router can front several DynamoGraphDeployments, such as production and canary graphs or two graphs serving the same model name in one Kubernetes namespace. Each graph is one Dynamo namespace (DYN_NAMESPACE), and its dynamo.frontend --router-mode direct resolves pinned instance IDs only within its own graph. Watching multiple namespaces therefore requires routing each pinned request to the matching graph's frontend pool.

Configuration and identity rules across multiple namespaces:

• Namespace lists: Set discovery.etcd.namespaces for etcd or discovery.kubernetes.dynamo_namespaces for Kubernetes sources (mutually exclusive with the scalar field, up to 16 unique entries; the Kubernetes namespace itself stays single).
• Per-namespace shim pools: When more than one namespace is configured, dynamo.shim.pools ([{namespace, backends}]) is required instead of shim.backends. Every configured namespace must have exactly one pool, and a backend replica may belong to at most one pool. Under kubernetes_pods, models entries must also specify dynamo_namespace.
• Worker identity: Every worker is identified by (namespace, instance_id) across reducer joins, duplicate checks, and first_seen tracking. The namespace is always taken from the watched etcd prefix, metadata path prefix, or nvidia.com/dynamo-namespace Pod label, never from an untrusted payload field.
• Graph-local dispatch: The router filters each namespace's shim pool by visibility and per-key policy, offers only namespaces with a usable replica to the picker, and keeps prefill/decode pairs within a single namespace. The load-only picker compares decode candidates across eligible namespaces first and picks prefill within the winning namespace; the KV-aware picker scores prefill candidates across namespaces that also have a decode candidate and picks decode within the winning namespace.

Kubernetes Pod source¶

On Kubernetes, the Dynamo operator sets DYN_DISCOVERY_BACKEND=kubernetes, so there is no etcd cluster. When the router's ServiceAccount can read pods (get, list, watch) but lacks RBAC for dynamoworkermetadatas and endpointslices, use dynamo.discovery.source: kubernetes_pods.

dynamo:
  enabled: true
  discovery:
    source: kubernetes_pods
    kubernetes:
      namespace: "dynamo"                       # Kubernetes namespace of the fleet's Pods
      dynamo_namespace: "dynamo-solar-pro4"     # the workers' DYN_NAMESPACE label value
      model: "solar-pro4"                       # served model name the frontend advertises
      watch_reconnect_backoff:
        initial: "500ms"
        max: "30s"
  shim:
    backends: [dynamo-frontend]

What the source reads¶

• Membership: Pods in namespace matching nvidia.com/dynamo-discovery-backend=kubernetes,nvidia.com/dynamo-discovery-enabled=true and nvidia.com/dynamo-namespace=<dynamo_namespace> (or in (a,b) with dynamo_namespaces).
• Role: Reads nvidia.com/dynamo-component-type (prefill or decode), or nvidia.com/dynamo-sub-component-type when the component type is worker (otherwise aggregated). Non-worker values (frontend, planner, epp, default) are skipped.
• Readiness: A Pod is included only while status.phase == "Running", condition Ready == "True", and metadata.deletionTimestamp is unset.
• Instance ID and Model: Derives instance_id from the Pod name (below) and assigns the configured model or per-component models mapping.

Because Pods carry no model cards or event channels, this source provides no total_kv_blocks, max_num_batched_tokens, kv_cache_block_size, dp_rank (no x-dynamo-dp-rank header is sent), or kv_metrics load signal. The picker rotates round-robin across candidates, config validate logs a warning, and GET /admin/dynamo/workers reports "signals": {"load": false, "capacity": false}.

Several models in one Dynamo namespace¶

When one Dynamo namespace hosts multiple worker pools serving different models, map each pool's nvidia.com/dynamo-component label to its served model using models instead of model:

dynamo:
  discovery:
    source: kubernetes_pods
    kubernetes:
      namespace: "kbm-g-np-kt-a"
      dynamo_namespace: "kbm-g-np-kt-a-solar-pro4-4tp-agg"
      models:
        - component: VllmWorker-decodeheavy
          model: solar-pro4
        - component: VllmWorker-prefillheavy
          model: solar-pro4-longcontext

Ready worker Pods match by exact, case-sensitive nvidia.com/dynamo-component value:

• Unlisted components or Pods without nvidia.com/dynamo-component stay held in memory without publishing (logged once at debug on readiness; not counted as malformed or refused).
• Using a single model across multi-model pools would mix them into one rotation, and dynamo.shim.model_map cannot split one Pod set into two models.
• Pods annotated with nvidia.com/dynamo-kube-discovery-mode: container are refused once and counted under continuum_dynamo_discovery_events_total{kind="refused"}.

Instance id derivation¶

Upstream derives a worker's instance ID from its Pod name via hash_pod_name (lib/runtime/src/discovery/kube/utils.rs in Dynamo 1.4.2 and 1.5.0): rustc's DefaultHasher::new() over the name as a str, masked to 53 bits (SipHash-1-3 with zero keys over the UTF-8 bytes plus 0xFF, & 0x001F_FFFF_FFFF_FFFF). The router implements SipHash-1-3 directly in src/dynamo/kube/hash.rs and pins it against live 1.4.2 /health IDs and upstream test vectors (worker-0 -> 1186445869714687).

Because Rust's DefaultHasher algorithm is not guaranteed stable across compiler releases, a unit test checks src/dynamo/kube/hash.rs against the router toolchain's DefaultHasher on every build. If a future Dynamo release is built with a compiler that changes DefaultHasher, the frontend will refuse unknown IDs with 500 Failed to generate completions. The registry stores IDs in lowercase hex, the shim sends decimal, and the Admin API displays both.

List, watch, and relist¶

Each pass builds a Kubernetes client on Tokio's blocking pool (with interactiveMode: Never so a hung kubeconfig exec plugin cannot block async workers), tries in-cluster credentials before KUBECONFIG / ~/.kube/config, lists matching Pods in pages of 500 up to the 16384-Pod ceiling, publishes them, and opens a watch at the list's resourceVersion. The connect, list, and watch-open steps share a 30s timeout; an abandoned client build is awaited by the next pass rather than spawned twice.

Watch lifecycle and backoff rules:

• Watches request a 290s server-side close (timeoutSeconds=290) and enforce a 320s per-event idle deadline so half-open connections cannot freeze connected at 1.
• Any watch termination (410 Gone, transport error, early EOF, or routine 290s close) triggers a full relist after the shared backoff rather than resuming from the last resourceVersion.
• The backoff resets to its floor after a watch session lasts at least min(watch_reconnect_backoff.max, 290s). Routine 290s closes do not increase the backoff or drop continuum_dynamo_discovery_connected from 1.
• At most 16384 Pods are held, and two ready Pods whose names hash to the same ID are both refused as duplicate.

RBAC¶

The router requires only get, list, and watch on pods in the fleet's namespace. Give the router a dedicated ServiceAccount rather than default.

In Helm, setting dynamoPodDiscovery.enabled: true renders the Role and RoleBinding in dynamoPodDiscovery.namespace, mounts the ServiceAccount token, and (when NetworkPolicy is enabled) permits egress on dynamoPodDiscovery.apiServerPorts (443 and 6443; set to [] and use networkPolicy.additionalEgress with an ipBlock to restrict destinations). For Kustomize, use deploy/kubernetes/overlays/dynamo-pod-discovery/.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: continuum-router-dynamo-pod-reader
  namespace: dynamo            # the fleet's namespace
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]

When to prefer the metadata-backed source¶

Use kubernetes_pods only when RBAC prevents reading dynamoworkermetadatas and endpointslices. Whenever an administrator can grant those resources, prefer dynamo.discovery.source: kubernetes (below), which reads model cards and event channels to enable capacity- and load-aware selection.

Reload¶

dynamo.discovery.source and dynamo.discovery.kubernetes reload live through the discovery supervisor:

• Switching source stops the running watcher, empties the registry, and starts the new source (reason="source").
• Editing namespace or dynamo_namespace resets the registry (reason="namespace"), editing model or models resets with reason="other", and editing dynamo_namespaces with unchanged kept-namespace models prunes only removed namespaces (reason="namespaces").
• Validation requires namespace to be a DNS-1123 label, dynamo_namespace a non-empty label value, and exactly one of model (non-empty, no whitespace) or models (1 to 256 unique component entries) when source: kubernetes_pods.

Kubernetes metadata source¶

dynamo.discovery.source: kubernetes mirrors Dynamo's own Kubernetes discovery daemon by reading DynamoWorkerMetadata custom resources joined with EndpointSlice readiness. Because DynamoWorkerMetadata embeds the same records an etcd fleet registers, the router feeds them directly into the etcd reducer, yielding full model cards, capacities, and kv_metrics / kv-events publishers without needing kubernetes.model.

dynamo:
  enabled: true
  discovery:
    source: kubernetes
    kubernetes:
      namespace: "dynamo"                       # Kubernetes namespace of the fleet
      dynamo_namespace: "dynamo-solar-pro4"     # the workers' DYN_NAMESPACE
      watch_reconnect_backoff:
        initial: "500ms"
        max: "30s"
  shim:
    backends: [dynamo-frontend]

What the source reads¶

The wire rules match ai-dynamo/dynamo v1.4.2 and v1.5.0 (src/dynamo/kube/metadata/):

• Resources: Lists and watches dynamoworkermetadatas.nvidia.com (v1alpha1) in namespace with no label selector. Each resource is named after and owned by its worker Pod, and spec.data holds endpoints, model_cards, event_channels, and event_sources keyed by etcd key suffixes (with null maps treated as empty).
• Readiness: Watches EndpointSlices in namespace matching nvidia.com/dynamo-discovery-backend=kubernetes,nvidia.com/dynamo-discovery-enabled=true. A DynamoWorkerMetadata resource is fed to the reducer only while an endpoint in a selected slice has conditions.ready: true and targetRef.name equal to the resource's name.
• Records: Each map entry becomes an etcd reducer event under v1/instances/, v1/mdc/, or v1/event_channels/, filtered to dynamo_namespace. Oversized card_json blobs (>1 MiB) are trimmed at classification time to their typed subset so held resources stay compact, and GET /admin/dynamo/workers reports "signals": {"load": true, "capacity": true}.
• Instance ID: Every endpoint and model-card record in a resource must carry the hash_pod_name ID derived from the resource name in both its path and value, or the entire resource is refused once as continuum_dynamo_discovery_events_total{kind="mismatch"}. Event channels are exempt because their instance_id is the publisher ID.
• Model: Read from each worker's card (display_name, then name). Setting kubernetes.model or kubernetes.models logs a validation warning and is ignored.

Container mode¶

Upstream's container discovery mode (DYN_KUBE_DISCOVERY_MODE=container) names resources <pod>-<container> and tracks readiness per container. This source supports pod mode only: any resource whose owner Pod name differs from the resource name is refused once and counted under kind="refused" (a main container collapses to the Pod's own name and is accepted with Pod-level readiness).

Feeding the reducer¶

A single etcd reducer lives for the watcher's lifetime and receives incremental diffs:

• When a resource becomes ready, its records are inserted; when it becomes not-ready or is deleted, its keys are removed.
• On resource updates, only modified records within the touched resource are re-fed. On EndpointSlice updates, only affected endpoints are toggled. Full relists re-put all records like an etcd resync.
• Reference counting across resources ensures shared event-channel keys leave the reducer only when their last contributing resource leaves, re-putting a surviving contributor's value if one departs.
• If an updated resource has a non-object spec or spec.data, it is counted once as malformed and retains the last valid records from the same metadata.uid, matching upstream daemon behavior.

Transport addresses¶

In this mode, an endpoint record's transport holds the worker's Pod IP and port. Just like etcd host:port addresses, it is kept internal and never exposed by GET /admin/dynamo/workers or Prometheus labels.

List, watch, relist, and bounds¶

Each pass shares a 30s seed timeout to connect, paginate DynamoWorkerMetadata (pages of 32) and EndpointSlices (pages of 500) up to 16384 objects each, and open both watches at their final resourceVersion before folding and publishing.

Both watches request a 290s server close and track independent 320s absolute idle deadlines so a stalled slice watch is detected even while the resource watch is active. When either watch ends, both streams perform a full relist after the shared backoff. Held resources are capped at 16384 objects and 256 MiB total record bytes (overflows are refused and counted once as malformed), on top of the reducer's 16384-key, 16 MiB record, and 1 MiB card bounds.

RBAC¶

The router requires get, list, and watch on dynamoworkermetadatas.nvidia.com and endpointslices.discovery.k8s.io (plus pods in the shared Role). Enable with dynamoPodDiscovery.enabled: true and dynamoPodDiscovery.metadata: true in Helm, or apply deploy/kubernetes/overlays/dynamo-metadata-discovery/ in Kustomize. Under a NetworkPolicy, also allow egress to the workers' kv_metrics publisher ports.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: continuum-router-dynamo-pod-reader
  namespace: dynamo            # the fleet's namespace
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["nvidia.com"]
    resources: ["dynamoworkermetadatas"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["get", "list", "watch"]

Reload¶

Reload rules match kubernetes_pods, except that edits to kubernetes.model or kubernetes.models under source: kubernetes are no-ops that neither restart the watcher nor empty the registry.

Fixtures¶

Unit tests in src/dynamo/kube/metadata/state_tests.rs use fixtures synthesized from pinned upstream sources (src/dynamo/kube/fixtures/*_synthesized.json) around four captured Pod names and verify that they produce the exact same fleet view as the equivalent etcd fixture.

Load signals¶

Discovery reveals which workers exist, while a second channel reports how busy each worker is. Every dynamo.vllm worker publishes an ActiveLoad sample on Dynamo's event plane whenever its load changes. The router subscribes to each publisher over ZMQ (or through an In-Cluster Event Broker when running outside the cluster) and retains the latest sample per worker rank (src/dynamo/load/, pinned to ai-dynamo/dynamo commit ff3ac59e83c73e03b98a5d0ec192ec28847130f7).

The event-channel key¶

Publishers register under the third watched etcd prefix:

Endpoint-scoped publishers (EventPublisher::for_endpoint) write keys of the form v1/event_channels/namespace/<ns>/component/<comp>/endpoint/<ep>/topic/kv_metrics/<publisher id as hex> (component- and namespace-scoped keys are also accepted). Segments use Dynamo's percent-encoding where ASCII alphanumerics, -, and _ pass through and other bytes become %XX (for example, ns.one -> ns%2Eone).

{
  "type": "EventChannel",
  "scope": {
    "kind": "endpoint",
    "endpoint": { "namespace": "dynamo", "component": "decode", "name": "generate" }
  },
  "topic": "kv_metrics",
  "instance_id": 2046539,
  "transport": { "kind": "Zmq", "config": { "endpoint": "tcp://10.0.0.7:41000" } }
}

Channel filtering rules:

• instance_id is the publisher incarnation's ID (publisher_id), not the worker's instance ID.
• Only direct Zmq channels with tcp:// endpoints become subscription targets.
• ZmqBroker, Nats, unknown transport/scope kinds, missing IDs, or non-tcp:// endpoints are skipped, logged once per key, and counted as unsupported or malformed.
• Unrecognized topics (active_sequences_events) are skipped silently, and duplicate publisher IDs are both refused as duplicate.

The topic filter, the frames, and the envelope¶

In direct ZMQ mode, the SUB socket subscribes to the bare topic kv_metrics, which matches ZMQ frame 0 and the envelope's topic field. The dotted EventScope::subject(topic) (namespace.<ns>.component.<comp>.endpoint.<ep>.kv_metrics) is used upstream only for NATS and XSUB broker mode; the router retains it for diagnostics, never as the ZMQ SUB prefix filter.

Each ZMQ message consists of four frames:

[ topic bytes ][ publisher_id: u64 BE ][ sequence: u64 BE ][ frame ]
frame = version: u8 (1) | payload_len: u32 BE | payload (payload_len bytes)

The payload decodes as a msgpack EventEnvelope { publisher_id, sequence, published_at (Unix ms), topic, payload: bytes }, whose inner payload is an ActiveLoad msgpack map (accepting both named-map and positional-array encodings across the N-2 window):

Non-fatal frame drops are counted under continuum_dynamo_load_events_total{result} without closing the subscription:

• malformed: Not four frames, ID frames not 8 bytes, or invalid msgpack.
• unknown_version: Frame header version is not 1.
• topic_mismatch: Frame 0 or envelope topic does not equal kv_metrics.
• oversized: Declared payload exceeds 16 KiB (refused before parsing).
• identity_mismatch: Frame and envelope publisher IDs disagree, publisher ID differs from the discovery record, or worker_id is not in the live registry snapshot.
• over_capacity: Load store is at its 65536-entry limit.

Dedup and ordering¶

Messages are deduplicated per subscription by sequence: any sequence at or below the highest applied value on that ordered TCP stream is dropped as duplicate. The load store applies the same sequence check per (worker_id, dp_rank) key, while allowing a sample from a different publisher_id (after a publisher restart) to replace the held entry regardless of sequence number.

Staleness and the best-effort contract¶

The load store records the receipt timestamp for each (worker_id, dp_rank) sample. fresh(worker_id, dp_rank, max_age) returns the sample only while its age is within max_age, and fresh_default uses dynamo.load.max_staleness (default 3s, range 100ms to 5m, hot-reloadable immediately).

Because upstream publishers are change-driven (debounced at 1ms, silent when load is constant) and best-effort, the picker reads a worker with no fresh sample as follows:

• Quiet (never published, or the last sample reported no backlog): Counts as idle (zero backlog) and competes on the same cost as the sampled workers, because a worker that goes idle publishes zero and falls silent.
• Silent under load (the last sample reported a backlog): May be a stalled engine, so it has no cost and is not chosen while another worker has one.
• No fresh sample on any worker: The load stream itself is silent or disconnected, so the per-phase round-robin cursor decides across all workers (round_robin_stale).

The router also counts the requests it pinned to each worker since that worker's last sample:

• Unanswered dispatches: Each request selected through the shim adds its prompt tokens (prefill) and blocks (decode) to the chosen workers' backlog until any sample from that worker arrives, for at most 30 seconds. A burst between two samples, or a quiet worker that never publishes (for example, because its own subscription is down), therefore fills up to its peers' load instead of taking every request.
• Stalled worker: A worker that leaves a request of two or more blocks unanswered for 5 seconds may be stalled. It is not chosen while another worker has a cost, until a sample arrives or the request ages out. Two requests do not count: one sent to the decode worker of a prefill and decode pair, which receives it only after the prefill finishes, and one that has already ended (refused by the frontend, failed, or cancelled), which may never have reached the engine. When no worker has a fresh sample, both pickers ignore a stall, since nothing could have answered the request.
• Missed messages: A kv_metrics subscription drops the samples it holds from its publisher whenever it may have missed a change, because a missed change (such as a busy worker going idle) would otherwise keep the worker out:
  • each time it connects, including the first connection of a subscription restarted by a transport switch or a changed publisher record;
  • whenever it sees a sequence gap.

On every registry generation change, the load supervisor atomically updates the store's live-worker admission gate and evicts samples and gauge series for departed workers (set_live_and_retain).

Subscriptions follow the registry¶

The load supervisor runs one SUB task per discovered publisher:

• Lifecycle: Appearing publishers start a task; departing or modified publishers have their tasks cancelled concurrently with a 5s grace period before abort. Tasks that fail to connect or read reconnect with full-jitter exponential backoff (500ms to 30s).
• Bounds: The load store is capped at 65536 (worker, dp_rank) entries, and active subscriptions are capped at 1024 sockets (admitted in ascending publisher_id order) so fleet-authored records cannot exhaust process file descriptors.

The bounded transport¶

Because fleet-authored tcp://ip:port endpoints are untrusted and release builds run under panic = "abort", the router implements its own bounded ZMTP 3.x SUB client (src/dynamo/load/zmtp.rs) over tokio::net::TcpStream instead of using zeromq 0.6 in production (which pre-allocates wire-declared 64-bit frame lengths without a ceiling).

Compiled-in transport bounds checked before buffer allocation:

Memory and connection behavior:

• Grow-on-demand buffers: Frame buffers start at 512 bytes and grow by powers of two up to MAX_FRAME_BYTES, paired with a 2 KiB read buffer and 64-byte write buffer (2,624 bytes per idle subscription, or ~2.6 MiB for 1024 subscriptions).
• Protocol scope: Supports only ZMTP 3.x NULL security, READY as SUB, and ZMTP 3.0 single-frame subscriptions (0x01 + topic). Violations count as continuum_dynamo_load_events_total{result="transport_refused"} and trigger backoff reconnects, resetting the backoff floor only after a 30s session.
• DNS resolution: Hostname endpoints resolve on Tokio's blocking pool under a 10s CONNECT_TIMEOUT; keep cluster DNS responsive if publishers announce hostnames rather than IP literals.

Load metrics¶

Per-worker gauge labels are bounded by fleet size (namespace is one of at most 16 configured namespaces, worker is lowercase hex, and dp_rank is decimal). nil fields export no series, and departed workers have their series removed. GET /admin/dynamo/workers exports only the load_publishers count, never internal tcp:// addresses.

Routing through a direct-mode shim¶

For every chat completion routed to a Dynamo fleet, the router selects the prefill and decode workers and dispatches the request to a dynamo.frontend --router-mode direct shim that executes the two-phase handoff.

Why a shim¶

dynamo.vllm workers speak only Dynamo's token-in request-plane protocol and do not support disaggregated serving over HTTP text endpoints. Running dynamo.frontend --router-mode direct provides Dynamo's external-picker contract: the frontend accepts an OpenAI HTTP request with x-dynamo-* worker-pin headers and handles tokenization, the 1-token prefill hop, disaggregated_params injection, the decode hop, detokenization, and SSE streaming.

Each shim replica is configured as an ordinary OpenAI-wire backend (role: unified) and listed under dynamo.shim.backends (or dynamo.shim.pools). Standard backend health checks, circuit breakers, rate limits, hub credentials, and per-key allowed_backends policies apply unchanged.

Request flow¶

1. The disaggregated orchestrator (disaggregated_serving.enabled: true) evaluates both streaming (stream: true) and buffered requests.
2. When dynamo is compiled and enabled and at least one permitted shim replica is available, the picker runs over candidate namespaces that have a usable shim replica. Client model IDs map through dynamo.shim.model_map for worker candidate matching only; the outbound request body keeps the client's model ID so the frontend can resolve it.
3. The picker selects a decode (instance, dp_rank) from decode or aggregated workers and, when the chosen worker is a decode worker, a prefill (instance, dp_rank) from the prefill workers of its namespace. When no prefill worker is chosen (an aggregated decode worker, or no prefill worker for the model), only the decode worker is pinned (aggregated fallback). The rules for aggregated workers beside a prefill/decode pool are under Mixed namespaces. If no worker serves the model, the request falls through to standard disaggregated or unified routing.
4. Any client-supplied x-dynamo-* headers are stripped, and the picker's headers are written on the outbound request to the chosen namespace's shim pool.
5. The shim's buffered or streaming SSE response is relayed through the standard pipeline with Continuum routing headers attached.

Outbound header contract¶

Header security and formatting rules:

• Decimal lease IDs: Both instance-id headers use decimal strings because dynamo.frontend treats hexadecimal IDs as absent. Rank headers are sent (including 0) whenever data_parallel_size is declared.
• Unconditional client header stripping: All client-supplied x-dynamo-* headers are stripped in always-compiled code across every build and configuration: at the global API middleware (src/http/middleware/dynamo_shim_headers.rs), at the chat ingress, and immediately before shim dispatch. Clients can never bypass policy by pinning workers directly.
• Response header disclosure: No x-dynamo-* header is echoed to the client. Worker and backend identities are disclosed only when the client sends an x-backend request header (even if blank):

Selection¶

The default LoadBasedPicker (src/dynamo/picker.rs) mirrors Dynamo's load-only cost formulas:

• prefill cost = active_prefill_tokens / max_num_batched_tokens
• decode cost = (active_decode_blocks + ceil(estimated_prompt_tokens / block_size)) / total_kv_blocks

Cost evaluation and tie-breaking:

• estimated_prompt_tokens uses the router's coarse character/word heuristic without a tokenizer. active_decode_blocks falls back to kv_used_blocks when nil (as on live Dynamo 1.4.2 dynamo.vllm workers). Workers with data_parallel_size > 1 contribute one candidate per dp_rank.
• A candidate's cost needs the capacity fields its card carries. A sample younger than dynamo.load.max_staleness supplies the backlog, and a quiet candidate (see Staleness and the best-effort contract) has zero backlog. Any other candidate without a fresh sample carrying the phase's field (for example, the nil active_prefill_tokens of dynamo.vllm) has no defined cost.
• When at least one candidate has a fresh sample carrying the phase's field, the lowest defined cost wins, and equal costs rotate on a per-phase round-robin cursor. Candidates with undefined costs are not chosen.
• When none has (for example, an idle fleet whose last samples aged past max_staleness, a disconnected load stream, or a dynamo.vllm prefill pool), the per-phase round-robin cursor rotates across all candidates.

Every selection increments continuum_dynamo_selection_total{role, reason} (role in prefill, decode):

Mixed namespaces¶

Dynamo serves no pair of a prefill worker and an aggregated worker, so the picker never sends x-dynamo-prefill-instance-id together with an aggregated worker's instance ID. Both pickers apply the same rules per namespace and model:

• Namespace with prefill and decode workers: Its aggregated workers are not decode candidates. Requests pair a prefill worker with a decode worker.
• Any other namespace: decode and aggregated workers are decode candidates.
• aggregated decode winner: No prefill worker is chosen, even when the namespace holds prefill workers. The selection is counted as {role="prefill", reason="no_prefill_pool"}.

When a published fleet view first makes a namespace serve one model from aggregated workers beside prefill or decode workers, the registry logs one warning with the namespace, the model, and the worker count per role. Dynamo 1.5.0 dynamo.frontend --router-mode direct cannot serve such a fleet: it picks one worker set at random per request and refuses an instance ID from the other. The warning is not repeated while the namespace stays mixed.

One publish names at most 8 (namespace, model) pairs. Model names come from etcd, so any further pairs that start mixing in the same publish are summarized in a single warning with the counts of suppressed and shown pairs.

DynamoWorkerPicker is pluggable via registry.picker() / set_picker(), allowing the KV-aware picker to replace the default without modifying AppState.

Failure semantics¶

• Connection retries: Shim connection failures may retry on another replica in the pool with the same worker-pin headers within the retry budget; workers are never re-selected for the same request.
• Mid-stream fallback: Standard mid-stream fallback rules apply after streaming starts without re-selecting workers.
• Direct-mode prefill refusal (503): When a direct-mode frontend refuses a buffered request with Prefill worker ID required in Direct routing mode (DIRECT_MODE_PREFILL_ID_ERROR in src/proxy/disaggregated_dynamo.rs), the router returns 503 (disaggregated_prefill_decode_unavailable), increments disaggregated_errors_total{phase="orchestration"}, and does not retry with another worker pair. If an upstream release wraps the error in a generic 500 body, it takes the decode failure arm below.
• Other shim failures: Counted under disaggregated_errors_total{phase="decode"}, followed by unified fallback when disaggregated_serving.fallback_to_unified is true (without instance-id headers) or disaggregated_decode_failed. On streaming requests, only pre-stream transport failures, 5xx, 429, or 408 trigger fallback; other 4xx responses are relayed directly.
• Phase histograms: disaggregated_prefill_duration_seconds and sibling phase histograms remain unwritten because phase boundaries occur inside the shim; disaggregated_requests_total{routing_path} is counted once per request.

Anthropic Messages and Responses ingresses¶

The Anthropic Messages ingress (/anthropic/v1/messages and /v1/messages) and the OpenAI Responses ingress (/v1/responses) also pin workers when dispatching to a shim backend (src/proxy/disaggregated_ingress.rs):

• Converted chat dispatches (pin_converted_chat_dispatch): Immediately before sending a converted /v1/chat/completions body, the ingress runs the chat path's worker selection narrowed to the chosen backend on the final post-rewrite body (including global_prompts and system-message merges) and attaches x-dynamo-* headers and nvext fields.
• Per-send selection: A fresh pin is chosen for every send, including each round of router-managed web_search / web_search_20250305 loops and each retry or fallback attempt. Client-supplied x-dynamo-* headers and nvext.token_data are always stripped.
• Native /v1/responses passthrough (pin_responses_passthrough_dispatch): When a shim replica is configured with type: openai or responses_api: native (or serves a responses_only model from the Anthropic ingress), passthrough /v1/responses bodies are pinned by model alone using x-dynamo-* headers without nvext.token_data.
• Error mapping and known gaps: Direct-mode refusals on buffered sends and non-2xx streaming heads return 503 (disaggregated_prefill_decode_unavailable on /v1/responses, or an Anthropic api_error ending with that code on /v1/messages). Refusals delivered inside an already-opened 200 SSE stream or over a Unix-socket streaming backend are relayed as-is, and POST /v1/responses/compact is not pinned.

Configuration¶

disaggregated_serving:
  enabled: true
  fallback_to_unified: false

backends:
  - name: dynamo-frontend
    type: openai
    url: "http://dynamo-frontend.dynamo.svc:8000"
    role: unified
    models: ["Qwen/Qwen3-32B"]   # an id the frontend can resolve

dynamo:
  enabled: true
  discovery:
    etcd:
      endpoints: ["http://etcd.dynamo.svc:2379"]
  shim:
    backends: [dynamo-frontend]

shim.backends, shim.pools, and shim.model_map hot-reload immediately without restarting watchers or emptying the registry. See docs/shared/llm-skills/config-templates/dynamo-direct-shim.yaml for a runnable template.

KV-overlap-aware prefill selection¶

Behind the opt-in dynamo-kv Cargo feature (included in official release binaries) and dynamo.kv.enabled: true, the router indexes block-level kv-events from workers and credits prefill candidates (or aggregated candidates, when the model has no prefill/decode pool) for prompt prefix blocks they already hold in device KV cache (src/dynamo/kv/, pinned to ai-dynamo/dynamo v1.5.0).

To match worker cache state byte-for-byte while keeping memory bounded under panic = "abort":

• Block hashing: Uses upstream dynamo-kv-hashing 1.6.1 directly.
• Radix index and cost formula: Implemented locally in src/dynamo/kv/ with strict memory ceilings rather than pulling dynamo-kv-router (whose indexer is unbounded and can panic on malformed fleet events).
• Tokenization and templates: Uses pure-Rust tokenizers and minijinja directly instead of dynamo-tokenizers / dynamo-renderer (avoiding C/C++ onig_sys, esaxx-rs, and hf-hub dependencies).

What is indexed¶

Workers started with --kv-events-config (dynamo.vllm) or --publish-kv-events (TensorRT-LLM) register a kv-events channel under v1/event_channels/… (GET /admin/dynamo/workers counts them under kv_event_publishers). The KV supervisor subscribes via the bounded ZMTP client using ZmtpLimits::KV_EVENTS (1 MiB per frame to fit batches of up to 128 events or 8192 blocks) under continuum_dynamo_kv_subscriptions{state} and the 1024-subscription cap.

Each RouterEvent { worker_id, storage_tier, event: { event_id, dp_rank, data } } updates the device-tier radix index (src/dynamo/kv/index.rs; non-device tiers increment other_tier):

Index structure and compiled-in bounds:

• Prefix matching: Nodes are keyed by tokens_hash from the root, with a per-rank lookup from engine block_hash (ExternalSequenceBlockHash) to node (ref-counted per rank in case two engine hashes name the same tokens_hash node). find_matches walks from the root and counts each rank's longest contiguous prefix match.
• Capacity ceilings: Bounded by MAX_INDEX_BLOCKS (1,048,576 nodes), MAX_BLOCKS_PER_RANK (262,144), MAX_INDEX_OWNERSHIPS (8,388,608 total (rank, block) ownerships), MAX_OWNERS_PER_NODE (4096 ranks), and 4096 (worker, dp_rank) cursors per subscription. Blocks past any ceiling are refused as over_capacity.
• Live identity gate: Only events for a live worker_id and a dp_rank declared by its card are admitted (identity_mismatch otherwise), updated atomically via set_live_and_retain when workers or ranks depart.

Gaps, and why there is no snapshot recovery¶

event_id is monotonic per publisher. The handler tracks a cursor per rank: events at or below the cursor are skipped as replays, while a forward jump clears that rank's blocks and increments continuum_dynamo_kv_events_total{result="gap"} so the rank rebuilds cleanly from subsequent root-anchored chains.

Unlike upstream's KV router, Continuum Router does not query worker_kv_query_source_<publisher id> over Dynamo's two-part TCP request plane after a gap. Clearing and rebuilding keeps the router off the internal request plane, and a temporary loss of overlap credit simply falls back to load-based selection.

The tokenizer dependency and its cache¶

Under dynamo.kv.enabled, the router resolves tokenizers and chat templates from the CheckedFile { path, checksum: "blake3:<64 hex>", size } slots on the workers' model cards (tokenizer, prompt_formatter, and chat_template_file), never from static router config:

• URI resolution and content-addressed cache: Local/file:// paths are always read, http(s):// worker metadata URLs are fetched when kv.allow_http_artifacts: true (default true), and hf:// URLs are fetched from Hugging Face (HF_ENDPOINT / HF_TOKEN) only when kv.allow_hf_artifacts: true (default false). Artifacts over 64 MiB are refused, verified against their blake3 checksum on download and on every read from kv.artifact_cache_dir/blobs/<checksum> (owner-only), and evicted if modified on disk.
• Mandatory operator pinning (kv.pinned_artifacts): Because model cards and Jinja templates are fleet-authored code, the router fetches, caches, loads, or renders an artifact only when its blake3 checksum is listed in kv.pinned_artifacts (empty by default). Unpinned artifacts increment continuum_dynamo_kv_artifacts_total{result="unpinned"} and continuum_dynamo_kv_selection_total{result="artifact_unpinned"} and log the checksum once per model for b3sum review. Removing a pin on reload drops the loaded tokenizer.
• Background loading and fleet agreement: The KV supervisor loads pinned tokenizers on the blocking pool (keyed by ArtifactKey, up to 16 cached tokenizers, 2 concurrent loads, 256 MiB total artifact bytes, 5m retry on failure). A request uses the tokenizer only when all workers serving the model agree on the same ArtifactKey; otherwise (such as during a rolling tokenizer upgrade or before loading finishes) it scores without a KV term (tokenizer_unavailable).

Prompt rendering and block hashing bounds:

• Bounded rendering on the blocking pool: Renders apply the first shim replica's global_prompts and requires_single_leading_system rewrites, then run the Hugging Face chat template (minijinja + minijinja-contrib) and tokenizers encoding on the blocking pool under a per-core permit (4 to 32 permits, 20ms permit wait, 2s total deadline, 5,000,000 minijinja fuel instructions, no file loader, 1 MiB rendered text cap, and 1,048,576 token cap). Exceeding limits falls back with prompt_too_large or render_unavailable.
• XXH3-64 block hashing: Computes dynamo-kv-hashing Request::block_hashes (XXH3-64 over little-endian u32 token IDs of complete blocks, seeded with 1337, lora_name, and the request's cache_salt) for up to two valid kv_cache_block_size values (16..=65536) per model. Unsupported block sizes log once and increment continuum_dynamo_kv_artifacts_total{result="unsupported"}.

The formula¶

KvAwarePicker (src/dynamo/kv/picker.rs) evaluates upstream's worker_logit formula for a candidate rank with block size bs and prompt length isl tokens:

overlap_blocks        = blocks of the request's prefix the rank holds
cached_tokens         = min(overlap_blocks * bs, isl)
raw_prefill_tokens    = active_prefill_tokens + (isl - cached_tokens) + cached_tokens
raw_prefill_blocks    = raw_prefill_tokens / bs
overlap_credit_blocks = overlap_score_credit * overlap_blocks
adjusted_prefill      = max(0, raw_prefill_blocks - overlap_credit_blocks)
decode_cost_blocks    = active_decode_blocks + max(0, ceil(isl / bs) - overlap_blocks)
cost                  = prefill_load_scale * adjusted_prefill + decode_cost_blocks

Selection rules:

• At kv.router_temperature: 0.0, the lowest cost wins with ties broken by a deterministic per-phase cursor; above 0.0, candidates are sampled via softmax_sample_index.
• When any candidate rank has a fresh load sample (< dynamo.load.max_staleness) carrying the phase's backlog, those ranks and the quiet ranks (zero backlog) compete on the full formula, and the other ranks sit out. When no rank has a fresh prefill backlog (as on dynamo.vllm fleets where active_prefill_tokens is nil), all ranks compete on overlap and request size (isl / bs - overlap_credit_blocks), rotating via round_robin_stale only on ties.
• With prefill workers, decode selection uses the load-only rule unless conditional disaggregation is enabled.
• With no prefill worker for the model (an aggregated fleet), each aggregated worker prefills its own requests, so the formula above scores the aggregated candidates. Their decode blocks (kv_used_blocks on dynamo.vllm) define the backlog, and active_prefill_tokens is added when a sample carries it.
• A prefill worker is a candidate only in a namespace that also holds a decode decode candidate, and the decode worker of a pair is always a decode worker (see Mixed namespaces). Prefill workers beside aggregated workers alone therefore leave the fleet scored as aggregated.
• Selections increment continuum_dynamo_selection_total{reason="kv_aware"} and continuum_dynamo_kv_selection_total{result} (overlap, no_overlap, tokenizer_unavailable, prompt_too_large, render_unavailable, artifact_unpinned, or bypass).

nvext.token_data¶

When kv.send_token_data: true (default false), the router attaches its rendered token IDs to the outbound shim body as nvext.token_data (unless shim pool replicas require different body rewrites), allowing dynamo.frontend to skip re-tokenizing. Any client-supplied nvext.token_data is always stripped before shim dispatch so clients cannot bypass prompt guardrails, and fallback copies sent to non-shim backends never carry nvext.token_data.

Conditional disaggregation¶

kv.conditional_disaggregation implements upstream's --router-conditional-disagg, scoring decode workers with max(0, decode_cost_blocks - overlap_credit_blocks) and deciding per request whether the chosen decode worker should prefill locally instead of using a separate prefill worker:

If decode_busy_threshold is set, the decode worker's projected active decode blocks must also be at or below decode_busy_threshold * total_kv_blocks (a missing sample or capacity vetoes the bypass). A bypass sends only x-dynamo-worker-instance-id, sets x-continuum-routing-path: unified, and counts reason="conditional_bypass" and result="bypass".

Note that on Dynamo 1.5.x, dynamo.frontend --router-mode direct rejects requests without a prefill ID whenever prefill workers are registered (503 disaggregated_prefill_decode_unavailable), so conditional_disaggregation.enabled defaults to false and config validate warns when it is enabled.

Configuration¶

dynamo:
  enabled: true
  discovery:
    etcd:
      endpoints: ["http://etcd.dynamo.svc:2379"]
  shim:
    backends: [dynamo-frontend]
  kv:
    enabled: true
    allow_http_artifacts: true    # a worker's own metadata URL (default)
    allow_hf_artifacts: false     # the Hub: public egress, opt in explicitly
    pinned_artifacts:             # what the router may load; nothing until pinned
      - "blake3:<64 hex of the reviewed tokenizer.json>"
      - "blake3:<64 hex of the reviewed tokenizer_config.json or chat_template.jinja>"
    overlap_score_credit: 1.0
    prefill_load_scale: 1.0
    router_temperature: 0.0
    send_token_data: false
    conditional_disaggregation:
      enabled: false

Every dynamo.kv field hot-reloads immediately without restarting discovery or emptying the registry. See docs/shared/llm-skills/config-templates/dynamo-kv-aware.yaml.

KV metrics¶

Every label set is closed, and continuum_dynamo_selection_total{reason} gains kv_aware and conditional_bypass.

In-cluster event broker listener¶

When Continuum Router runs outside the Kubernetes cluster, it can often watch the API server for discovery but cannot dial worker Pod IPs for kv_metrics or kv-events. Enabling the dynamo-event-bus Cargo feature (included in official release binaries) and dynamo.event_bus.enabled: true starts a dedicated inbound HTTP/2 over TLS listener that in-cluster continuum-event-broker and continuum-kv-listener agents dial out to.

The tunnel¶

Built on the shared crates/continuum-event-bus wire contract, a broker or listener opens two types of HTTP/2 requests to the router's listener:

• POST /bus/v1/session (control channel): A long-lived bidirectional NDJSON stream. The client sends hello (role, client_id, reachable Dynamo namespaces or announced KV sources, and supported protocol versions), and the router replies with welcome (sess-<32 hex> and effective limits). The router then sends subscribe and unsubscribe commands, the client sends stream_closed and draining, and both sides exchange heartbeat / heartbeat_ack.
• POST /bus/v1/streams/{stream_id} (data stream): Opened by the client for each subscribe command to carry the publisher's raw ZMTP or SSE bytes.

The router never dials the broker; every connection is inbound to the listener.

What the listener enforces¶

The listener binds its own port (dynamo.event_bus.listen) and rejects any configuration that collides with a TCP server.bind_address.

• TLS 1.3 and HTTP/2 (h2) only: Uses rustls with the ring provider. The listener peeks at the first byte of every TCP connection: anything other than a TLS handshake (0x16) is refused as tls_required (writing a best-effort plaintext HTTP/1.1 403 JSON body if the byte is an ASCII letter). Handshakes must complete within 10s (handshake_timeout or handshake_failed), and HTTP/2 keepalive PINGs run every 20s.
• Authentication: Validates Authorization: Bearer <token> in constant time against brokers[].token_hash (never storing plaintext tokens), or verifies a client certificate chaining to tls.client_ca whose DNS SAN matches a broker id (lowercase, no wildcards). When both are sent, the token decides and the certificate must not name a different broker (forbidden). Missing or unknown credentials return 401 unauthenticated, and credentials removed on reload return 401 credential_revoked (256-entry ring).
• Certificate CA binding: Certificate-authenticated identities stay bound to the CA generation that verified the TLS connection.
• hello validation: Requires header continuum-bus-version >= 1 (400 unsupported_protocol_version) and a hello message within 10s. Only one pre-hello handshake per broker credential can occupy admission at a time (too_many_sessions for another), preventing a stalled client from exhausting global session slots. The hello role (broker or kv_listener) and client_id must match the credential's brokers[] entry (forbidden).
• Namespace and backend allowlists: A broker hello may declare only Dynamo namespaces in its entry's namespaces allowlist (forbidden; an empty allowlist permits any namespace but logs a validation/startup warning). A kv_listener hello must announce at least one source, no dynamo_namespaces, and only backends listed in its backends grant (forbidden).
• Session and connection caps: Allows one active session per configured entry (a new session supersedes the old one with session_superseded), up to max_sessions welcomed sessions router-wide (429 too_many_sessions), and up to 4 * max_sessions concurrent TCP connections (minimum 16, connection_limit). A TLS connection must have a 2xx session or stream request accepted within 10s (auth_timeout, even if it answers HTTP/2 PINGs) and closes 30s after its last accepted request ends. Silent sessions end after idle_timeout.
• Commanded streams only: Accepts data streams only for unconsumed stream IDs commanded in the same session (404 unknown_stream, 409 duplicate_stream), authenticated by the owning broker's credential (403 forbidden), at the negotiated protocol version.

Byte bounds¶

Memory per session is bounded by max_session_buffer_bytes (default 64 MiB). Each stream's per-direction buffer is derived as max_session_buffer_bytes / (4 * max_streams_per_session) and clamped to 16 KiB..=1 MiB (16 KiB at defaults; configurations below max_streams_per_session * 64 KiB are rejected).

The four reservations per data stream cover the HTTP/2 receive window, the current received frame, the outbound pipe (payload plus queue-item overhead), and the HTTP/2 send buffer, while the HTTP/2 connection window equals max_session_buffer_bytes. The listener serves HTTP/2 directly via hyper to enforce these exact windows.

Hot reload¶

spawn_event_bus_supervisor applies configuration reloads without touching discovery watchers or the worker registry:

Listener startup failures log event bus listener failed to start; retrying and retry every 30s.

The command API¶

continuum_router::dynamo::event_bus::EventBus connects the listener supervisor to the load and KV subscription supervisors:

• EventBus::open_stream(target: ZmqTcpTarget, topic: Topic) -> Result<BusStream, BusStreamError>: Selects the highest-ranked live, non-draining broker session covering target's namespace that is below its stream cap, sends subscribe, and resolves to a cancel-safe BusStream (AsyncRead + AsyncWrite + Unpin + Send) within 30s (NoSession, TooManyStreams, Refused(code), OpenTimeout, SessionClosed, Busy, or InvalidTarget).
• EventBus::open_announced_stream(backend: &str, last_event_id: Option<u64>) -> Result<BusStream, BusStreamError>: Selects the least-loaded live, non-draining kv_listener session granted backend, sends subscribe with last_event_id, and returns a BusStream carrying SSE bytes.
• Stream lifecycle and state: Dropping a BusStream sends unsubscribe. session_count(), covered_namespaces(), covers(namespace), local_addr(), is_running(), and the subscribe_sessions() watch channel expose listener state.

Event bus metrics¶

Every label set is closed; no stream ID, session ID, broker ID, or client string appears in a label.

Configuration¶

dynamo:
  event_bus:
    enabled: true
    listen: "0.0.0.0:7443"
    tls:
      cert: "/etc/continuum-router/event-bus/tls.crt"
      key: "/etc/continuum-router/event-bus/tls.key"
      # client_ca: "/etc/continuum-router/event-bus/client-ca.crt"
    brokers:
      - id: "broker-a.dynamo"
        token_hash: "sha256:<64 lowercase hex>"   # printf %s "$TOKEN" | sha256sum; never the token
        namespaces: ["dynamo"]
      - id: "kv-listener-a"                    # role kv_listener: granted backends, not namespaces
        role: kv_listener
        token_hash: "sha256:<64 lowercase hex>"
        backends: ["vllm-1", "vllm-2"]
    max_sessions: 64
    max_streams_per_session: 1024
    max_session_buffer_bytes: 67108864
    heartbeat_interval: "10s"
    idle_timeout: "30s"

See the configuration guide and docs/shared/llm-skills/config-templates/dynamo-event-bus.yaml.

The kv_listener push role¶

A brokers[] entry with role: kv_listener serves continuum-kv-listener in push mode over the same protocol version 1 contract, adding AnnouncedSourceTarget.last_event_id (equivalent to SSE Last-Event-ID) and requiring at least one announced source in Hello::validate:

Runtime behavior for kv_listener entries:

• Per-backend grant: Shares TLS, authentication, and max_sessions limits with brokers, but authorizes commands per backend listed in backends. Narrowing backends or changing role on reload stops new placements and closes affected live sessions with forbidden.
• SSE consumption and drain: kv_cache_index sources with transport: event_bus call open_announced_stream through KvEventStreamOpener, parse the relayed SSE bytes, and resume from last_event_id. When a listener sends draining, the router immediately unsubscribes its open streams so consumers fail over to another session.
• Validation advisories: Warns when a transport: event_bus source is configured while dynamo.event_bus.enabled is false, when no kv_listener entry grants a configured bus source, or when a kv_listener grant has no matching bus source. See KV Cache Optimization.

Subscribing through a broker¶

With dynamo.load.transport: broker, kv_metrics and kv-events subscriptions run the router's bounded ZMTP client over event-bus streams instead of dialing publisher Pod IPs directly.

The connector¶

ZmtpConnector (src/dynamo/load/connector.rs) abstracts transport dialing:

• DirectTcpConnector dials tcp://host:port directly.
• BrokerConnector (src/dynamo/load/broker.rs, behind dynamo-event-bus) parses the publisher endpoint into a ZmqTcpTarget in the publisher's Dynamo namespace and calls EventBus::open_stream(target, topic).
• All ZMTP handshakes, frame/message ceilings, msgpack decoding, publisher-ID verification, sequence deduplication, and kv-events gap handling run in the router over the relayed bytes, with a 30s stream-open timeout, 10s connect budget, redacted ZmtpError::Broker reason strings, and a 1s CLOSE_TIMEOUT.

Pending¶

Before dialing, BrokerConnector checks EventBus::covers(namespace). When no live broker session covers the publisher's namespace, the subscription enters the pending state (its workers have no fresh sample) and waits on EventBus::subscribe_sessions() without polling. When a covering broker connects, waking subscriptions apply up to 500ms of uniform random delay (PENDING_WAKE_JITTER) before opening streams. kv-events subscriptions retain their per-rank cursors across session drops and reconnects.

Several brokers per namespace¶

Multiple broker sessions can cover the same Dynamo namespace:

• Rendezvous placement: EventBus::open_stream ranks live, non-draining covering sessions by 64-bit rendezvous hash SHA-256(len(key) || key || client_id) (where key encodes namespace, host, port, and topic) and picks the highest-weighted session below max_streams. Placement is deterministic across router replicas, avoids moving open streams when a broker joins, and spreads load evenly (1/N).
• Single-reconnect failover: When a broker dies, ZmtpConnector::carrier_lost detects the ended session within 500ms (SESSION_LOSS_GRACE), keeps the reconnect delay at its 500ms floor (RECONNECT_INITIAL), and immediately places the subscription on the next-ranked covering broker.
• Ageing cut-off kv-events subscriptions: If a kv-events subscription stays disconnected or pending for 60s (KV_EVENTS_STALE_AFTER), all ranks named by its cursors are cleared from the KV index and logged at warn so stale cache state does not skew prefix scoring.
• Uncovered namespace visibility: Under transport: broker, GET /admin/dynamo/workers includes broker_coverage, and a 250ms-debounced reporter exports continuum_dynamo_broker_namespace_sessions{namespace} and continuum_dynamo_broker_pending_subscriptions{family, namespace} for every configured namespace. Builds without dynamo-event-bus keep broker subscriptions in pending and never fall back to direct dialing silently.

Switching the transport¶

dynamo.load.transport hot-reloads immediately: supervisors stop running subscriptions concurrently and restart them over the new transport without restarting discovery or clearing the load store or KV index. Restarted kv-events subscriptions carry over their per-rank event_id cursors so any events missed during the switch trigger normal gap recovery, while permanently stopped subscriptions clear their ranks from the index.

Observability¶

GET /admin/dynamo/workers lists active subscriptions under subscriptions, and subscription gauges include transport (direct, broker) and state (pending, connecting, connected, backoff):

• continuum_dynamo_load_subscriptions{state, transport}
• continuum_dynamo_kv_subscriptions{state, transport}
• continuum_dynamo_broker_subscriptions{family, broker} (bounded by configured brokers[].id entries)

End-to-end test¶

tests/dynamo_load_broker_test.rs tests the broker in-process, while tests/dynamo_broker_tunnel_test.rs spawns the real continuum-event-broker binary against a fake Kubernetes API server and loopback zeromq::PubSocket publishers on every CI run:

cargo test --features "dynamo-kv dynamo-event-bus" --test dynamo_broker_tunnel_test

The test verifies three scenarios:

• Same state as a direct dial: Replaying kv_metrics and kv-events (including late sequences, foreign publisher IDs, removals, and gaps) over transport: direct and transport: broker yields identical load store and KV index state.
• Killing the broker: Sending SIGKILL to the broker moves subscriptions to pending (round_robin_stale selection); starting a new broker reconnects all subscriptions, restores load selection, and catches missed kv-events removals via gap clearing.
• Unadvertised publisher: Publishers omitted from DynamoWorkerMetadata are refused with target_not_advertised while advertised publishers connect.

Observability¶

Discovery exports three Prometheus metric families with closed label sets:

GET /admin/dynamo/workers¶

Mounted behind Admin authentication and audit middleware when both dynamo and admin features are compiled:

{
  "enabled": true,
  "connected": true,
  "source": "etcd",
  "signals": {"load": true, "capacity": true},
  "namespace": "dynamo",
  "namespaces": ["dynamo"],
  "generation": 12,
  "workers": [
    {
      "instance_id": "1a7f2b",
      "namespace": "dynamo",
      "worker_id": "1736491",
      "role": "decode",
      "model": "Qwen/Qwen3-32B",
      "component": "decode",
      "endpoint": "generate",
      "dp_start_rank": 2,
      "dp_size": 4,
      "block_size": 64,
      "capacity": {
        "total_kv_blocks": 12000,
        "max_num_batched_tokens": 8192,
        "max_num_seqs": 256,
        "context_length": 32768
      },
      "disaggregated_endpoint": "dynamo/prefill/generate",
      "first_seen": "2026-09-23T04:11:02+00:00",
      "last_updated": "2026-09-23T04:19:48+00:00"
    }
  ],
  "load_publishers": 1,
  "kv_event_publishers": 1,
  "subscriptions": [
    {
      "family": "kv_metrics",
      "publisher_id": "2046539",
      "namespace": "dynamo",
      "component": "decode",
      "state": "connected",
      "transport": "broker",
      "broker": "broker-a.dynamo"
    },
    {
      "family": "kv-events",
      "publisher_id": "2046540",
      "namespace": "dynamo",
      "component": "decode",
      "state": "pending",
      "transport": "broker"
    }
  ]
}

Response fields and privacy guarantees:

• broker_coverage and subscriptions: Under dynamo.load.transport: broker, broker_coverage reports covered, brokers, and pending_subscriptions per configured namespace. subscriptions lists each running kv_metrics and kv-events subscription's state, transport (direct or broker), and carrying broker ID.
• namespaces, source, signals, and worker_id: namespaces lists configured namespaces (namespace is the first entry), source and signals describe the active discovery source, and worker_id is the decimal form of instance_id used by GET /health and x-dynamo-worker-instance-id.
• No internal addresses or raw cards: Worker transport addresses, Pod IPs, node names, UIDs, publisher tcp:// endpoints, and raw model cards are never exported.
• Disabled vs. uncompiled state: Returns 200 with "enabled": false when dynamo is compiled but disabled in live config, and 404 when built without the dynamo feature.

Module layout¶

Because the discovery reducer is a pure function of its event stream, src/dynamo/apply_tests.rs, apply_duplicates_tests.rs, apply_channels_tests.rs, and src/dynamo/kube/metadata/state_tests.rs verify materialization, duplicate handling, and Kubernetes-to-etcd equivalence without running an etcd or Kubernetes server.

Reading the registry from code¶

AppState holds dynamo_registry: Option<Arc<DynamoWorkerRegistry>>, which is always Some when built with the dynamo feature so live reloads can enable discovery at any time. Consumers check registry.is_enabled() rather than re-reading configuration:

• registry.snapshot(): Returns Arc<DynamoSnapshot> lock-free (snapshot.namespace identifies the snapshot's primary namespace).
• registry.subscribe(): Returns a tokio::sync::watch::Receiver<u64> carrying a monotonic change generation so slow subscribers cannot pin old snapshots in memory.
• registry.load_store(): Returns Arc<DynamoLoadStore> (fresh(worker_id, dp_rank, max_age) with worker_id resolved via worker_id_of_instance).
• registry.picker() / set_picker(): Reads or replaces the active Arc<dyn DynamoWorkerPicker>.

The snapshot is swapped before the generation counter increments, and idempotent resyncs that do not change the fleet view do not bump the generation.

Build requirement¶

The dynamo feature compiles etcd-client (which invokes protoc via tonic-prost-build). Any build enabling dynamo, dynamo-kv, or dynamo-event-bus (including cargo clippy --all-features) requires protoc on PATH:

brew install protobuf                  # macOS
apt-get install -y protobuf-compiler   # Debian/Ubuntu

Default (default/full) and embed builds compile none of these features and do not require protoc; official release binaries compile all three.

Compatibility¶

Wire contracts were verified against upstream ai-dynamo/dynamo, and end-to-end harness image tags are pinned in deploy/examples/dynamo-direct-shim/.env:

Live-fleet validation¶

The rows below record validation against a live Kubernetes fleet running ai-dynamo==1.4.2 using a read-only identity with get/list/watch on Pods and Services (dynamo.discovery.source: kubernetes_pods):

Divergences observed on the live 1.4.2 fleet:

• (a) vLLM chat processor drops the worker pin: Unpatched 1.4.2 --dyn-chat-processor vllm fails to copy nvext routing hints from x-dynamo-* headers on /v1/chat/completions (fixed for SGLang in ai-dynamo/dynamo#14696; live runs used a frontend image with the equivalent vLLM fix).
• (b) Multiple models in one Dynamo namespace: Pinning a worker from another model's pool returns 500 Failed to generate completions (instance_id=... not found). Use kubernetes.models to scope pools by component.
• © ActiveLoad carries only kv_used_blocks: Both active_decode_blocks and active_prefill_tokens arrive as nil on 1.4.2 dynamo.vllm workers, so decode cost uses kv_used_blocks and LoadBasedPicker rotates prefill workers round-robin (while KvAwarePicker ranks them by prefix overlap and request size).
• (d) Ephemeral publisher ports and decimal headers: Workers bind ZMQ PUB sockets on tcp://0.0.0.0:0 (OS-assigned ports recorded in discovery metadata), and the frontend requires decimal x-dynamo-worker-instance-id values (treating hex as absent 400).

Live check of the broker tunnel¶

To verify the event-bus tunnel against a live cluster (kbm-g-np-kt-a) with an external router:

• Run a router built with dynamo, dynamo-kv, and dynamo-event-bus outside the cluster with dynamo.load.transport: broker, dynamo.discovery.source: kubernetes (or etcd), and a reachable dynamo.event_bus listener covering the fleet's namespace.
• Deploy continuum-event-broker in the workers' Kubernetes namespace (deploy/helm/continuum-event-broker or deploy/kubernetes/event-broker) pointed at bus_url with get/list/watch on dynamoworkermetadatas.nvidia.com.
• Run the read-only verification script (--probe-image optionally runs a temporary non-root curl Pod to measure TCP+TLS relay latency):

ROUTER_ADMIN_TOKEN=... scripts/dynamo-broker-live-check.sh \
    --context <context> --namespace <broker namespace> \
    --router-url https://<router host>:<port> \
    --window 300 --max-staleness-ms 3000 \
    --probe-image curlimages/curl:8.11.1 --bus-url https://<bus host>:<bus port>

The script verifies three steps in order:

1. Pods reach the bus: Checks that every row in GET /admin/dynamo/workers subscriptions is connected over transport: broker, and inspects continuum_dynamo_event_bus_sessions and broker Pod metrics.
2. Load samples stay fresh: Scrapes /metrics once per second over --window to report decoded kv_metrics / kv-events frames, maximum inter-frame gap against dynamo.load.max_staleness, and the share of reason="load" vs. reason="round_robin_stale" decode selections under traffic.
3. Relay latency: When --probe-image is provided, times TCP+TLS handshakes from the broker namespace to --bus-url and reports half the median connect time as the one-way bus latency estimate.

What stays Dynamo-owned¶

Continuum Router chooses the prefill and decode workers, while the following remain entirely inside Dynamo:

• Tokenization and detokenization: Owned by dynamo.frontend (unless kv.send_token_data: true supplies pre-rendered token IDs).
• The prefill/decode handoff: The 1-token prefill hop, disaggregated_params injection, and decode hop over Dynamo's request plane.
• KV cache transfer: Direct block transfer between prefill and decode workers (such as NIXL).
• Worker liveness: Managed by each worker's own etcd lease or Kubernetes readiness.
• OpenAI response shaping and engine streaming: Produced by dynamo.frontend and relayed by the router.

Because prefill and decode execute inside the shim, disaggregated_prefill_duration_seconds and sibling phase histograms are not observable by the router and remain unwritten.

Testing against a real cluster¶

tests/dynamo_etcd_discovery_test.rs runs against a live etcd instance when ETCD_ENDPOINTS is set (writing only under a process-unique namespace and cleaning up on exit):

docker run --rm -d -p 2379:2379 --name etcd-test quay.io/coreos/etcd:v3.5.17 \
    etcd --advertise-client-urls http://0.0.0.0:2379 --listen-client-urls http://0.0.0.0:2379

ETCD_ENDPOINTS=http://127.0.0.1:2379 \
    cargo test --features dynamo --test dynamo_etcd_discovery_test

CI test coverage without external clusters:

• Watch session unit tests (src/dynamo/watcher_tests.rs): Drives drive_session through a scripted WatchSource to verify put/delete folding, compaction cancellation, full Resync recovery, and first_seen preservation on every CI pass.
• ZMQ load subscriber (tests/dynamo_load_zmq_test.rs): Binds a loopback zeromq::PubSocket and verifies sequence deduplication, max_staleness expiry,subscription teardown, and byte-exact decoding of captured 1.4.2 fixtures (src/dynamo/load/fixtures/kv_metrics_frame_captured*.bin).
• Shim dispatch (tests/dynamo_shim_picker_test.rs): Runs against a wiremock shim to verify x-dynamo-* decimal header injection, client header stripping, aggregated fallback, SSE relay, and 503 disaggregated_prefill_decode_unavailable classification.

For full end-to-end verification with real Dynamo processes, scripts/e2e-dynamo-shim.sh (.github/workflows/dynamo-e2e.yml, manual dispatch) brings up deploy/examples/dynamo-direct-shim/ (etcd, dynamo.mocker prefill/decode/aggregated workers, dynamo.frontend --router-mode direct, and the router).

On Dynamo 1.5.0, the frontend withdraws a disaggregated model when its last prefill worker leaves (needs: [["prefill"]]) and randomly splits traffic if decode and aggregated workers coexist, so aggregated workers must take over only after the disaggregated pool drains; the harness tests this exact transition.