Model Experiments (A/B Testing)¶

Model experiments let clients keep asking for one stable model name, such as chat, while the router sends a controlled share of that traffic to one upstream model and the rest to another. Each experiment is a virtual model name that resolves, per request, to one of several weighted variants. A variant names the upstream model id to dispatch and may pin the backends allowed to serve it.

Experiments work for hosted providers (OpenAI, Anthropic, Gemini, Bedrock) as well as self-hosted engines, because the rename happens in the router rather than on the engine.

A runnable starting point ships as continuum-router config generate --template ab-testing.

Configuration¶

model_experiments:
  - name: chat-ab-2026-09       # experiment name: sticky-hash salt and metric label
    model: chat                 # client-facing model name
    sticky_key: api_key         # api_key (default) | user | none | header:<name>
    sticky_ttl: 7d              # optional: re-assign each caller once per window (unset = permanent)
    fallback: within_variant    # within_variant (default) | cross_variant
    response_model: upstream    # upstream (default) | client
    response_headers: true      # emit attribution headers (default false)
    hide_variant_models: false  # hide and reserve the variant models (default false)
    variants:
      - id: control
        model: gpt-5.6          # upstream model id sent to the backend
        weight: 90
      - id: candidate
        model: qwen3.6-235b
        backends: [vllm-pool]   # optional backend pin
        weight: 10

Experiment fields¶

Variant fields¶

The section hot-reloads immediately: it is read from the current configuration on every request.

How a request is resolved¶

The experiment runs at the same point in every model-routed ingress where metadata alias dispatch runs: chat completions, completions, embeddings, the Responses API (including /v1/responses/compact), Anthropic Messages (including count_tokens), and the image endpoints. The order is:

1. model_aliases rewrites the name first (Anthropic and Responses ingresses), so an alias slot may point at an experiment's client-facing model.
2. Smart routing runs on the chat ingress. A tier target that is an experiment's client-facing model is resolved by the experiment. The smart routing virtual_model itself may not be an experiment model; load-time validation rejects that.
3. The request-parameter policy snapshot and the per-key model gate judge the client-facing name. Write request_params.models.<name> and a key's allowed_models against chat, not against the variant models.
4. The experiment picks a variant, and the request continues under the variant's upstream model.
5. Metadata alias dispatch runs on that upstream model, so a variant model that is itself a metadata alias still dispatches as its canonical id.

A request that already carries a routing decision the router must not alter skips the experiment: an AppProxy ingress pin, a Continuum Hub substitution, arbitrage, or an exact-model budget guard.

Selection errors raised after the rewrite (a 404 model not found or a 403 model not permitted) name the client-facing model, not the variant's upstream id.

Some model-routed ingresses do not resolve experiments: /v1/rerank, /embed_sparse, the /v1/realtime WebSocket, and /v1/batches dispatch the requested model name literally. An experiment's client-facing model is not served there, so send those requests with a concrete upstream model id.

Sticky assignment¶

A sticky caller lands in the same arm on every replica and across restarts. The router folds the experiment name (the salt) and the sticky value through SHA-256 and assigns the arm by weighted rendezvous hashing. Nothing depends on process-local randomness.

A request that does not carry its sticky value (no API key, no user field, an absent or blank header), and every request under none, is assigned by an independent random draw weighted the same way. These requests are counted with assignment="random" in model_experiment_requests_total, so an experiment whose split is dominated by random draws is visible.

Changing weights¶

Weighted rendezvous hashing gives each caller an independent draw per variant, so a weight edit moves only the callers the new weights require:

• raising one arm's weight only moves callers into that arm;
• lowering one arm's weight only moves callers out of that arm;
• adding, removing, or zeroing an arm only moves the callers that arm gains or loses.

Changing a 90/10 split to 80/20 moves roughly 10% of callers, all from control to candidate. To re-run an experiment with a fresh assignment, rename it.

Assignment lifetime¶

By default a sticky caller keeps its arm for as long as the experiment exists. sticky_ttl re-assigns each caller once per window, still without storing anything: the window index is folded into the caller's sticky hash, so every replica computes the same arm for the same caller at the same wall-clock time, and a restart changes nothing.

• Fixed, staggered windows. Each caller's windows are sticky_ttl long and shifted by an offset derived from that caller's own sticky hash, so callers re-roll at different moments across the window rather than all at once. A window is not measured from the caller's first request.
• A re-roll is a fresh draw. At a boundary a caller lands in the same arm again with probability equal to that arm's weight share, so the share of callers that change arm at each re-roll is 1 - sum(share^2): 18% for a 90/10 split, 50% for 50/50.
• Conversations can switch arms. A conversation that spans a caller's boundary continues on the newly drawn arm.
• Weight changes still move callers minimally within a window.
• Changing sticky_ttl reshuffles. A new TTL changes both the offsets and the window indices, so it re-assigns callers immediately; like the rest of the section it takes effect on the next request.
• No effect without a sticky value. With sticky_key: none, or for a request that carries no sticky value, every request is already an independent draw; config validate warns when sticky_ttl is set with sticky_key: none.

The window index is not part of the response-cache key: the key already carries the assigned variant.

Backend pins¶

A variant's backends list narrows the backend allow-list the request carries. It is the same list a key's allowed_backends restriction feeds into, so the pin bounds selection, retries, the response-cache route, and every fallback hop. When the key's own backend allow-list and the pin share no backend, the request is refused with 403 model not permitted rather than being moved to another arm.

Failure semantics¶

Backend pins still apply to fallback hops under cross_variant, so a pinned variant that should be able to fall back must list the fallback target's backends too.

Cache isolation¶

The assigned experiment and variant are folded into the response-cache namespace shared by the exact, prefix, and semantic caches. One arm never answers from the other arm's entries, even when both arms dispatch the same model through the same backend. Requests that no experiment resolved keep their cache keys unchanged.

Response model name¶

With client, a JSON body larger than 32 MiB and a single SSE line larger than 16 MiB are passed through without the rewrite.

Use response_headers: true when clients or log pipelines need to attribute a response to an arm without exposing it in the body.

Model listing¶

/v1/models, /v1/models/extended, and /v1/models/{model} list the experiment's client-facing name while at least one positive-weight variant model is served. The entry's backends are the union of the backends serving each variant model, narrowed to each variant's pin, and the entry goes through the same per-key backend and model visibility filters as any other model.

/anthropic/v1/models lists the client-facing name the same way, with the id as its display name.

Hiding and reserving variant models¶

With hide_variant_models: true, clients can reach the variant models only through the experiment. Every declared variant model, including one with weight 0, is both hidden and reserved.

Hidden¶

The variant models are removed from /v1/models, /v1/models/extended, /v1/models/proxy, and /anthropic/v1/models, and /v1/models/{model} answers 404 for them. The experiment's client-facing name stays listed, and its entry still takes its backends from the variant models before they are removed. The Admin API model catalog (/admin/models) and the WebUI are operator surfaces and keep showing the variant models.

Reserved¶

A request that names a variant model outside the experiment is refused with the same 404 model not found the ingress returns for a model the router does not have, naming the model the client sent. The response body is identical to an unknown model's, so the refusal does not reveal that the model exists. The check runs on the name the per-key model gate judges, after model_aliases, on chat completions, completions, embeddings, the Responses API, Anthropic Messages and count_tokens, the image endpoints, /v1/rerank, /embed_sparse, and /v1/realtime. On chat completions and completions it runs on the name the client sent (after an AppProxy ingress pin, which replaces that name), so a Hub decision cannot lift it. The refusal never comes earlier than an unknown model's answer would: a request error (a bad file reference, a missing rerank query) and a per-key allowed_models 403 are returned first, exactly as for an unknown name, and on chat completions a name that smart routing replaces (intercept_all) is routed to a non-reserved model like any other name.

• Aliases do not bypass it. A request for an exact metadata alias of a reserved model, or for the canonical id of a reserved alias, is refused too.
• Per-key allowed_models does not lift it. Listing a variant model in a key's allowed_models does not make it reachable.
• Experiment traffic passes. The experiment's own variant rewrite, and fallback: cross_variant hops along a chain that starts at a reserved model, still dispatch reserved models.
• Other routing never lands on one. Smart routing removes reserved models from its candidates, and a fallback chain whose origin is not reserved skips its reserved hops at runtime, so an ordinary model's chain never reaches one.
• Hub decisions are trusted. A Continuum Hub substitution, arbitrage, or budget fallback onto a reserved model is operator-authored routing, like a chain that starts at a reserved model, and is not refused. The client-facing name it rewrites from is still checked.
• Experiment responses still name the variant. The default response_model: upstream returns the variant's upstream model in the response model field. Set response_model: client when clients should see only the experiment's client-facing name.
• Batches are not covered. /v1/batches forwards an uploaded input file to the operator's batch backend and never sees the per-line model names, so a batch line can still name a reserved model.

Each refusal increments model_experiment_reserved_refusals_total{experiment}.

A hidden experiment may not use a variant model as its own client-facing model, because hiding would reserve the name clients are told to send; the loader rejects that combination. continuum-router config validate warns (and the loader logs) when a reserved model is also a model_aliases target, a target of a fallback chain from a non-reserved model, a smart_routing.model_profiles entry, or a variant of another experiment that does not hide its variants, which keeps it reachable through that other experiment.

Metrics¶

Experiments use a dedicated metric family rather than adding labels to the existing request, latency, and token families, which would change their label sets and multiply their cardinality. Every label value comes from configuration or a closed set, and at most 32 experiments with 16 variants each can be configured.

The duration is measured to the response headers, which for a streaming request is the start of the stream. The status class is the HTTP status the router answered with.

Token-counting requests (/anthropic/v1/messages/count_tokens) resolve the experiment, so they count on the assigned variant's model and honor its backend pin and emit the attribution headers, but they are not counted in model_experiment_requests_total or model_experiment_request_duration_seconds. Clients that count tokens before every call would otherwise inflate the per-arm request count used as the denominator below.

# Error rate per arm
sum by (experiment, variant) (rate(model_experiment_requests_total{status="server_error"}[5m]))
  / sum by (experiment, variant) (rate(model_experiment_requests_total[5m]))

# Completion tokens per request, per arm
sum by (variant) (rate(model_experiment_tokens_total{kind="completion"}[5m]))
  / sum by (variant) (rate(model_experiment_requests_total{status="success"}[5m]))

When OTLP trace export is enabled, the router.request span carries the assignment as router.experiment and router.experiment_variant (see Trace Export). Usage records sent to Continuum Hub do not carry the experiment or variant yet; that is tracked with the usage-ledger work in #1624.

Validation¶

The loader rejects, at startup, hot reload, the Admin API apply path, and continuum-router config validate:

• duplicate experiment names, and a client-facing model used by two experiments;
• an experiment model equal to an enabled smart_routing.virtual_model;
• an experiment with no variants, or with every weight 0;
• a weight above 1000000, a duplicate variant id, or a name or id outside [A-Za-z0-9._-]{1,64};
• a backend pin naming an unknown or internal: true backend;
• a variant model that none of its pinned backends lists, when every pinned backend has an explicit models list;
• a variant model that is another experiment's client-facing model (experiments do not chain);
• an invalid sticky_key, or unknown fields;
• a sticky_ttl that is not a whole number with an optional s, m, h, or d suffix, or is outside 60s to 365d;
• a hide_variant_models: true experiment with a variant whose model equals the experiment's own client-facing model;
• a configured backends[].name equal to the identifier the router reserves internally for refusing a request whose backend pin and API-key allow-list share no backend. The identifier embeds a NUL byte, so no real backend name can collide with it by accident; the check exists only to keep that refusal path from ever matching a configured backend.

Troubleshooting¶