Model Variant Suffixes: Design Record¶

Model variant suffixes are an opt-in, per-request selection surface: a client appends :nitro, :floor, :thinking, or :online to a model name and the router applies that preference to the request's own dispatch (issue #2405). Without the switch nothing changes.

Configuration¶

Set proxy.variant_suffixes.enabled: true. The default is false.

The grammar is interpreted on /v1/chat/completions only, buffered and streaming. Every other ingress, including /v1/completions, /v1/responses, and the Anthropic Messages routes, keeps its existing behavior: a suffixed name there is an unknown model. The switch reloads immediately, because the chat handler parses the name from the same configuration snapshot it dispatches with.

Resolution order¶

The full requested name is resolved first, through the same layers /v1/chat/completions already uses: a declared id or alias in an enabled backend's model_configs, the layered model metadata cache, a configured models: entry, or a live catalog entry with backends. Only when that fails does the router look for a recognized trailing suffix, splitting at the last : and resolving the base id.

Three consequences follow, and all three are covered by tests:

A name a backend serves literally keeps resolving. An Ollama name:tag is never misread as a variant.
A suffix on a base the router cannot resolve stays an unknown model. The error names the full requested name as before.
An unrecognized trailing word is never split. base:cheap and base:free fail as unknown models. :free and :extended, which some clients use elsewhere, are not implemented.

Suffix behavior¶

Suffix	Behavior
`:nitro`	Narrows the healthy, admissible candidates to the servers tied at the highest fresh `generation_throughput_tps`, then the configured strategy chooses among the tie.
`:floor`	Narrows them to the servers tied at the cheapest estimated cost for this request, using the same pricing metadata and request cost estimate as `least_cost` selection, with the output rate as tiebreak, then the configured strategy chooses among the tie.
`:thinking`	Raises an unset `reasoning_effort` to `high` on a model whose metadata declares thinking support. A client-supplied effort wins.
`:online`	Runs the existing router-managed `web_search` loop.

nitro and floor narrow candidates, they never override selection. Health filtering and the engine-saturation admission gate run before the narrowing, so a preference can only choose among servers that could have served the request anyway. A ranking needs data: a server with no fresh engine-stats snapshot makes no throughput statement for :nitro, and a server with no comparable price is excluded from :floor. When no candidate can be ranked, the set is left whole, normal selection serves the request, and the response reports the preference as not applied.

Refusals¶

Two variants can be unsatisfiable, and both refuse rather than silently serve the base model:

:thinking on a model whose metadata declares neither a thinking pattern nor non-empty reasoning_efforts.
:online with no usable executor: web_search disabled or absent, a commercial backend that the router never injects into, or a streaming request, which the buffered web-search loop cannot serve.

The answer is HTTP 404 with error.type: "invalid_request_error" and error.code: "unsupported_model_variant", and the message names the model and the suffix. :nitro and :floor never refuse.

Identity and accounting¶

The base id is what the request is for. Per-key allowed_models, the per-model rate limit, and the per-model token budget all key on the base id, so a suffix cannot escape a limit configured for the model, and a key scoped to the base may use its variants. The backend receives the base id.

Exact-first applies to these gates too. A name that is itself a served or declared model keeps its own identity even when its tail is a suffix word, so a model literally served as base:nitro is admitted and charged as base:nitro, never as base.

The suffix is preserved where a client or operator needs to see it:

the response's top-level model is rewritten to the requested name, for both a JSON body and the data lines of a stream;
x-continuum-model-variant names the variant word (nitro, floor, thinking, or online);
x-continuum-model-variant-applied: false appears only when a :nitro or :floor preference was evaluated and could not be applied;
continuum_model_variant_requests_total{variant, applied} counts requests by variant and by whether the preference took effect.

A variant belongs to the request's own primary attempt. A fallback-chain hop dispatches a different model, so it never inherits the suffix. An x-backend preference wins the first attempt as it always does, which bypasses the candidate ranking and reports the variant as not applied.

Limits¶

The variant does not change the response cache identity. A request for base:nitro and one for base may therefore share a cache entry when routing has already narrowed the candidates to one backend; the cache identity is the route, not the requested spelling. This applies to :nitro, :floor, and :thinking.
:online never reads or writes the response cache, on the exact, prefix, and semantic paths. The key is built from the payload before the web_search tool is injected, so a cached entry would answer the request without searching, and a deployment with no usable executor must still refuse instead of replaying a cached answer.
Streaming :online is refused rather than served without search, because the router-managed loop is buffered only.
:thinking sets a reasoning effort and does nothing else. It does not enable a provider-specific thinking budget the model metadata does not already describe.

Open questions¶

Whether :free and :extended, which the issue also lists, map to anything in a self-hosted fleet. Neither is implemented.
Whether the grammar should cover an ingress other than /v1/chat/completions, such as /v1/completions or the Responses API.