Skip to content

Per-Request Provider Routing Preferences: Design Record

This note records the design for client-expressed per-request routing preferences, for parity with OpenRouter's provider object (issue #2396). The feature shipped behind the proxy.client_provider_preferences switch; the user-facing contract is in the API reference under Client Provider Preferences, and this page records how the design was settled and what changed from the original plan.

Today

What exists alongside the shipped feature:

  • x-backend header: a client can pin one backend per request on every model-routed endpoint (BACKEND_PREFERENCE_HEADER in src/proxy/selection.rs; see the API reference). It remains a first-attempt pin and wins over provider.
  • Passthrough: with the switch off (the default) the chat handler never reads a provider key, so under the backend-passthrough contract an unknown top-level field reaches local and generic OpenAI-compatible backends unchanged (only the strict cloud OpenAI and Gemini paths strip unknown fields). A client's provider object therefore still reaches an aggregator backend such as OpenRouter, where it is meaningful.
  • backends[].request_extensions.body: per-backend defaults and overrides, merged leaf by leaf into the client's object. With the switch on, an operator can still send a provider object to a backend that understands it through overrides.provider, which is applied at dispatch after the client key is removed.

Shipped design

  • Config surface: proxy.client_provider_preferences, default false, interpreting the client's provider object on /v1/chat/completions dispatch. Absent or disabled, nothing changes: the provider key keeps passing through as it did before.
  • Parsing and limits: the raw value is kept as serde_json::Value and validated by hand rather than by deny_unknown_fields, so every refusal can name the exact path (provider.<field>, provider.max_price.<key>, a list index). The serialized object is capped at 16 KiB, each name list at 16 entries, each name at 1 to 128 bytes with no leading or trailing whitespace and no control characters, and duplicates collapse preserving first occurrence.
  • Interpreted fields: order (a soft rank list), only / ignore (hard allowlist and blocklist), sort (price, throughput, latency), and max_price (per-million-token USD ceilings on prompt and completion). require_parameters and data_collection are accepted syntactically but only in their permissive form; the restrictive values are refused by name. Unknown fields are refused with 400 rather than ignored.
  • Security: the candidate set is built from backend visibility, the key's allowed_models and allowed_backends, Hub and policy narrowing, and only then the client's filters. The client's only/ignore can never widen a key's allowlist or a routing policy: the intersection is taken, never the union. A filter that empties a non-empty candidate set answers 400, so an unsatisfiable preference is a client error and never a silent switch to a backend the client excluded.
  • Body hygiene: with the switch on, the router consumes and removes the provider object before dispatch, so a backend that would reject the unknown field fails nothing. Dropping it changes passthrough to aggregator backends, which is why the switch is opt-in; an aggregator backend opts back into forwarding with request_extensions.body.overrides.provider.
  • Deduplication: a request carrying preferences never joins another request's in-flight dispatch. The deduplication key hashes the dispatched body, and the object is already gone from it, so two requests differing only in their preferences would otherwise coalesce and one could receive an answer from a backend it excluded. Requests without the object coalesce exactly as before.
  • Telemetry seam for sort: price uses the same estimated-request-cost helper the cost reporting uses, throughput uses the fresh engine-stats snapshot, and latency uses the pool's recorded average response time. A candidate missing the stated signal joins one trailing rank group, so a fleet with no telemetry yields a single group and the configured selection_strategy decides, unchanged from before.
  • Grouping: candidates equal on every stated key form one rank group and the configured strategy rotates within it, so equal prices or equal order ranks do not pin the first candidate.

Settled questions

  • Default sort: none. Omitting sort and order means the configured selection_strategy applies unchanged. There is no operator-side field restriction beyond the switch itself; the strict field set and the refused restrictive values are the restriction.
  • Aggregator backends: with the switch on, the client object no longer reaches any backend verbatim. An operator forwards a provider object deliberately through request_extensions.body.overrides.provider, which wins over the removed client key, so static configuration has the last word. defaults.provider cannot restore the client object because defaults only fill unset keys.
  • order and fallback: order re-sorts the surviving candidates for each attempt rather than pinning one. Smart-routing, quota, and health filters narrow first; order and sort then rank what is left, and retries after the first attempt apply the same preferences again. x-backend still wins the first attempt only.
  • Fallback on refusal: a provider refusal maps to a 400 validation error, which is not a fallback trigger by default. On the streaming path the fallback chain is also gated off for this refusal, because a 200 from another model is a different answer, not a fallback.
  • Anthropic ingress: a top-level provider key on /anthropic/v1/messages is refused while the switch is on instead of being silently dropped, since that ingress does not apply client routing preferences.