Model Experiments (A/B Testing)¶

Model experiments let clients keep asking for one stable model name, such as chat, while the router sends a controlled share of that traffic to one upstream model and the rest to another. Each experiment is a virtual model name that resolves, per request, to one of several weighted variants. A variant names the upstream model id to dispatch and may pin the backends allowed to serve it.

Experiments work for hosted providers (OpenAI, Anthropic, Gemini, Bedrock) as well as self-hosted engines, because the rename happens in the router rather than on the engine.

A runnable starting point ships as continuum-router config generate --template ab-testing.

Configuration¶

model_experiments:
  - name: chat-ab-2026-09       # experiment name: sticky-hash salt and metric label
    model: chat                 # client-facing model name
    sticky_key: api_key         # api_key (default) | user | none | header:<name>
    sticky_ttl: 7d              # optional: re-assign each caller once per window (unset = permanent)
    fallback: within_variant    # within_variant (default) | cross_variant
    response_model: upstream    # upstream (default) | client
    response_headers: true      # emit attribution headers (default false)
    hide_variant_models: false  # hide and reserve the variant models (default false)
    variants:
      - id: control
        model: gpt-5.6          # upstream model id sent to the backend
        weight: 90
      - id: candidate
        model: qwen3.6-235b
        backends: [vllm-pool]   # optional backend pin
        weight: 10

Experiment fields¶

Variant fields¶

The section hot-reloads immediately: it is read from the current configuration on every request.

How a request is resolved¶

Model experiments run at the same stage as metadata alias dispatch on supported model-routed ingresses:

• Chat completions (/v1/chat/completions) and completions (/v1/completions)
• Embeddings (/v1/embeddings)
• Responses API (/v1/responses, including /v1/responses/compact)
• Anthropic Messages (/anthropic/v1/messages, including count_tokens)
• Image generation and editing endpoints

Resolution order¶

1. model_aliases rewrite: on the Anthropic and Responses ingresses, model_aliases rewrites the incoming model name first, allowing an alias slot to target an experiment's client-facing model.
2. Smart routing: on the chat ingress, smart routing runs next. If a tier target points to an experiment's client-facing model, the experiment resolves it. (The smart routing virtual_model itself cannot be an experiment model; load-time validation rejects that configuration.)
3. Policy and access gates: the request-parameter policy snapshot (request_params.models.<name>) and the per-key allowed_models gate evaluate the client-facing model name (for example, chat), not the underlying variant models.
4. Variant selection: the experiment selects a variant arm and rewrites the target model to the variant's upstream model.
5. Metadata alias dispatch: metadata alias resolution runs on the selected variant's upstream model, so a variant that references a metadata alias still dispatches to its canonical model ID.

Bypass conditions, error reporting, and excluded ingresses¶

• Pinned routing decisions: requests that already carry an authoritative routing decision skip experiment resolution. This includes AppProxy ingress pins, Continuum Hub model substitutions, arbitrage routing, and exact-model budget fallback guards.
• Error attribution: backend selection errors raised after variant resolution (404 model not found or 403 model not permitted) report the client-facing model name rather than the variant's upstream model ID.
• Excluded ingresses: /v1/rerank, /embed_sparse, the /v1/realtime WebSocket, and /v1/batches do not resolve model experiments and dispatch the requested model name literally. Call those endpoints with a concrete upstream model ID.

Sticky assignment¶

A sticky caller lands in the same arm on every replica and across restarts. The router folds the experiment name (the salt) and the sticky value through SHA-256 and assigns the arm by weighted rendezvous hashing. Nothing depends on process-local randomness.

A request that does not carry its sticky value (no API key, no user field, an absent or blank header), and every request under none, is assigned by an independent random draw weighted the same way. These requests are counted with assignment="random" in model_experiment_requests_total, so an experiment whose split is dominated by random draws is visible.

Changing weights¶

Weighted rendezvous hashing gives each caller an independent draw per variant, so a weight edit moves only the callers the new weights require:

• raising one arm's weight only moves callers into that arm;
• lowering one arm's weight only moves callers out of that arm;
• adding, removing, or zeroing an arm only moves the callers that arm gains or loses.

Changing a 90/10 split to 80/20 moves roughly 10% of callers, all from control to candidate. To re-run an experiment with a fresh assignment, rename it.

Assignment lifetime¶

By default a sticky caller keeps its arm for as long as the experiment exists. sticky_ttl re-assigns each caller once per window, still without storing anything: the window index is folded into the caller's sticky hash, so every replica computes the same arm for the same caller at the same wall-clock time, and a restart changes nothing.

• Fixed, staggered windows. Each caller's windows are sticky_ttl long and shifted by an offset derived from that caller's own sticky hash, so callers re-roll at different moments across the window rather than all at once. A window is not measured from the caller's first request.
• A re-roll is a fresh draw. At a boundary a caller lands in the same arm again with probability equal to that arm's weight share, so the share of callers that change arm at each re-roll is 1 - sum(share^2): 18% for a 90/10 split, 50% for 50/50.
• Conversations can switch arms. A conversation that spans a caller's boundary continues on the newly drawn arm.
• Weight changes still move callers minimally within a window.
• Changing sticky_ttl reshuffles. A new TTL changes both the offsets and the window indices, so it re-assigns callers immediately; like the rest of the section it takes effect on the next request.
• No effect without a sticky value. With sticky_key: none, or for a request that carries no sticky value, every request is already an independent draw; config validate warns when sticky_ttl is set with sticky_key: none.

The window index is not part of the response-cache key: the key already carries the assigned variant.

Backend pins¶

A variant's backends list narrows the backend allow-list the request carries. It is the same list a key's allowed_backends restriction feeds into, so the pin bounds selection, retries, the response-cache route, and every fallback hop. When the key's own backend allow-list and the pin share no backend, the request is refused with 403 model not permitted rather than being moved to another arm.

Failure semantics¶

Backend pins still apply to fallback hops under cross_variant, so a pinned variant that should be able to fall back must list the fallback target's backends too.

Cache isolation¶

The assigned experiment and variant are folded into the response-cache namespace shared by the exact, prefix, and semantic caches. One arm never answers from the other arm's entries, even when both arms dispatch the same model through the same backend. Requests that no experiment resolved keep their cache keys unchanged.

Response model name¶

With client, a JSON body larger than 32 MiB and a single SSE line larger than 16 MiB are passed through without the rewrite.

Use response_headers: true when clients or log pipelines need to attribute a response to an arm without exposing it in the body.

Model listing¶

/v1/models, /v1/models/extended, and /v1/models/{model} list the experiment's client-facing name while at least one positive-weight variant model is served. The entry's backends are the union of the backends serving each variant model, narrowed to each variant's pin, and the entry goes through the same per-key backend and model visibility filters as any other model.

/anthropic/v1/models lists the client-facing name the same way, with the id as its display name.

Hiding and reserving variant models¶

With hide_variant_models: true, clients can reach the variant models only through the experiment. Every declared variant model, including any variant with weight: 0, is both hidden from client model listings and reserved against direct requests.

Hidden¶

• Client model endpoints: variant models are omitted from /v1/models, /v1/models/extended, /v1/models/proxy, and /anthropic/v1/models, and /v1/models/{model} returns HTTP 404 for them.
• Experiment listing: the experiment's client-facing model remains listed, aggregating the backends of its positive-weight variants before the variant entries are hidden.
• Operator surfaces: the Admin API model catalog (/admin/models) and the WebUI continue to display the underlying variant models.

Reserved¶

Direct client requests for a reserved variant model outside the experiment are rejected:

• Indistinguishable 404 response: the router refuses the request with the same 404 model not found payload that the ingress returns for an unknown model, echoing the model name the client sent so the refusal does not reveal that the variant model exists.
• Covered ingresses: the reservation check runs on the post-model_aliases model name (the same name evaluated by the per-key model gate) across chat completions, completions, embeddings, the Responses API, Anthropic Messages and count_tokens, the image endpoints, /v1/rerank, /embed_sparse, and /v1/realtime. On chat completions and completions, it checks the client-supplied name (after any AppProxy ingress pin), ensuring a Hub routing decision cannot bypass the check on the incoming model name.
• Error and routing precedence: the refusal never fires earlier than an unknown-model 404 would. Request validation errors (such as an invalid file reference or a missing rerank query) and per-key allowed_models 403 errors are returned first. On chat completions, if smart routing (intercept_all) replaces the incoming name, the request routes to a non-reserved candidate model as usual.
• Aliases do not bypass it: requesting a metadata alias of a reserved model, or the canonical ID of a reserved alias, is also refused.
• Per-key allowed_models does not lift it: listing a variant model in a key's allowed_models does not make it directly callable.
• Experiment traffic passes: the experiment's own variant rewrite, as well as fallback: cross_variant hops along a fallback chain that starts at a reserved model, can still dispatch reserved models.
• Other routing never lands on one: smart routing excludes reserved models from its candidate pool, and fallback chains starting from a non-reserved model skip reserved hops at runtime.
• Hub decisions are trusted: a Continuum Hub substitution, arbitrage route, or budget fallback onto a reserved model is treated as operator-authored routing and is allowed to proceed (while the incoming client-facing name is still checked before substitution).
• Experiment responses still name the variant: by default (response_model: upstream), responses carry the variant's upstream model ID in the model field. Set response_model: client to expose only the experiment's client-facing name.
• Batches do not resolve experiments or reservations: /v1/batches does not rewrite experiment names or apply hide_variant_models reservations. It does check inline Anthropic models and restricted OpenAI input-file models against the effective Hub tier and local-key model allow-lists. A reserved variant can therefore appear in a batch only when those access lists permit it, or when the caller has no model restriction.

Each refusal increments model_experiment_reserved_refusals_total{experiment}.

Configuration validation for reserved models¶

• Self-referential variant rejected: an experiment with hide_variant_models: true cannot use its own client-facing model as a variant model (doing so would reserve the name clients are meant to call); config loading rejects that combination.
• Reachability warnings: continuum-router config validate warns (and the loader logs a warning) when a reserved model is also referenced as a model_aliases target, a fallback target of a non-reserved model, a smart_routing.model_profiles entry, or a variant of another experiment that has hide_variant_models: false.

Metrics¶

Experiments use a dedicated metric family rather than adding labels to the existing request, latency, and token families, which would change their label sets and multiply their cardinality. Every label value comes from configuration or a closed set, and at most 32 experiments with 16 variants each can be configured.

The duration is measured to the response headers, which for a streaming request is the start of the stream. The status class is the HTTP status the router answered with.

Token-counting requests (/anthropic/v1/messages/count_tokens) resolve the experiment, so they count on the assigned variant's model and honor its backend pin and emit the attribution headers, but they are not counted in model_experiment_requests_total or model_experiment_request_duration_seconds. Clients that count tokens before every call would otherwise inflate the per-arm request count used as the denominator below.

# Error rate per arm
sum by (experiment, variant) (rate(model_experiment_requests_total{status="server_error"}[5m]))
  / sum by (experiment, variant) (rate(model_experiment_requests_total[5m]))

# Completion tokens per request, per arm
sum by (variant) (rate(model_experiment_tokens_total{kind="completion"}[5m]))
  / sum by (variant) (rate(model_experiment_requests_total{status="success"}[5m]))

When OTLP trace export is enabled, the router.request span carries the assignment as router.experiment and router.experiment_variant (see Trace Export). On control-plane builds, every usage record for a request resolved by a model experiment carries experiment_id (model_experiments[].name) and experiment_variant_id (model_experiments[].variants[].id), independent of quality sampling.

Validation¶

The loader rejects, at startup, hot reload, the Admin API apply path, and continuum-router config validate:

• duplicate experiment names, and a client-facing model used by two experiments;
• an experiment model equal to an enabled smart_routing.virtual_model;
• an experiment with no variants, or with every weight 0;
• a weight above 1000000, a duplicate variant id, or a name or id outside [A-Za-z0-9._-]{1,64};
• a backend pin naming an unknown or internal: true backend;
• a variant model that none of its pinned backends lists, when every pinned backend has an explicit models list;
• a variant model that is another experiment's client-facing model (experiments do not chain);
• an invalid sticky_key, or unknown fields;
• a sticky_ttl that is not a whole number with an optional s, m, h, or d suffix, or is outside 60s to 365d;
• a hide_variant_models: true experiment with a variant whose model equals the experiment's own client-facing model;
• a configured backends[].name equal to the identifier the router reserves internally for refusing a request whose backend pin and API-key allow-list share no backend. The identifier embeds a NUL byte, so no real backend name can collide with it by accident; the check exists only to keep that refusal path from ever matching a configured backend.

Troubleshooting¶