Shadow Traffic¶

Shadow traffic mirroring replays a sampled share of live client requests to a second model without changing the response any client receives. It answers a question that model experiments cannot: how would a candidate model behave on the traffic that the primary model is already serving?

Model experiments split live traffic, so a candidate is evaluated only on the share it is allowed to serve. Shadow traffic instead sends every matched request to the primary model exactly as before, and sends a copy to the shadow model after the primary request has been dispatched. Both answers are produced from the same prompt, and only the primary one reaches the client.

Mirroring is opt-in and off by default. When shadow is absent or enabled: false, request handling is unchanged.

Configuration¶

shadow:
  enabled: true
  sample_rate: 0.05            # 0.0 to 1.0 of matching requests
  models: ["chat", "gpt-5.6-sol"]   # client-facing models to mirror (empty = all)
  target:
    model: gpt-5.6-luna        # model name resolved by the normal routing pipeline
    # backend: azure-eu        # optional pin; otherwise normal selection
  max_request_bytes: 65536
  max_concurrent: 8            # in-flight shadow requests; excess is shed
  timeout: 60s
  stream: false                # drain streamed shadow responses without accumulating
  headers: strip               # strip client auth and tenant headers; do not forward
  sample_key: api_key          # deterministic sampling key: api_key | user | none

What mirroring does and does not do¶

The mirror runs on a detached task after the primary request is dispatched. It changes nothing the client observes:

• The primary response, its status, and its headers are byte-identical to a run with the section absent.
• Client-visible latency is unaffected, because the mirror is never awaited in the request path.
• A shadow failure of any kind is recorded as a metric and never surfaces.

The mirror is also isolated from the router's own accounting:

• It does not consume client rate-limit quota. The mirror never re-enters the HTTP layer, so the client rate-limit middleware never sees it.
• It does not consume a per-model or per-backend rate-limit bucket. Selection runs through the same strategy pipeline but never charges the bucket that a dispatched request charges.
• It does not move the primary backend's circuit state in either direction. The mirror never records a success, a failure, or a timeout against it. Selection still honours an open circuit, so a mirror is not sent to a backend the router is currently avoiding.
• It writes no usage or statistics row.

Client headers¶

The mirror sends a sanitized header map. The shadow backend authenticates with its own configured credential, never with anything the client sent.

Every other client header is forwarded. There is no mode that forwards the stripped set. Header names are matched case-insensitively with _ read as -, so a client x_api_key is stripped like x-api-key.

Sampling¶

The sampling decision happens before anything is cloned, so an unmatched or unsampled request costs the mirror nothing.

• sample_key: api_key samples from a stable 64-bit hash of the authenticated API key id.
• sample_key: user samples from a stable 64-bit hash of the resolved user identity.
• sample_key: none samples randomly per request.

A deterministic key is stable across process restarts and across replicas, so a caller that samples in stays in for a given sample_rate. When no identity is available for the configured key, for example a router that runs without authentication, the decision falls back to a random draw rather than skipping the request.

After a request samples in, its body is measured before it is cloned; a sampled request over max_request_bytes is counted as skipped_size and not replayed.

Concurrency and timeout¶

Each mirror needs one of max_concurrent slots for the router. When every slot is taken, the mirror is shed and counted as shed: nothing is queued, so a slow shadow model cannot grow an unbounded backlog. A slot is taken when the mirror's attempt starts, not while the primary request is still running, and it is held until the body read finishes.

timeout bounds one attempt end to end. An expired mirror is counted as timeout and its connection is dropped.

Streaming¶

A streamed client request mirrors like a buffered one. When stream: true, the shadow request asks the backend for a stream and the reply is drained chunk by chunk on the detached task, so the router never holds a whole shadow completion in memory. When stream: false, the shadow request asks for a buffered reply and discards it after reading the token counts.

The client's own stream is a separate response and is never touched.

Metrics¶

Both model and backend come from configuration, so cardinality is bounded by the configured targets rather than by traffic. The families are registered at startup, so a router that never enables the section still reports them at zero.

Privacy¶

Prompt and completion content is never logged and never used as a metric label. The mirror's log line carries only the configured target model, the requested model name, the selected backend, whether the reply succeeded, and the latency. Header values are never logged.

Hot reload¶

The section is read from the configuration each request runs under, so a changed sample rate, target, concurrency cap, or timeout applies to the next request. Disabling shadow stops new mirrors immediately; a mirror already in flight may finish.

Related pages¶

• Model Experiments for splitting live traffic between variants.
• Metrics for the scrape endpoint and the metric families it exposes.
• Rate Limiting for the client and per-model quota the mirror deliberately bypasses.
• Circuit Breaker for the state the mirror leaves untouched.