Skip to content

Backend Passthrough Contract

For several backend types, Continuum Router forwards the chat completions request body without backend-specific reshaping once the request reaches the transport layer. This is narrower than an edge-to-upstream byte-equivalence guarantee: earlier handler stages may still modify the payload before make_http_request, make_unix_socket_request, or LlamaCppBackend::execute_chat_completion see it. This document defines which backends offer that transport-layer guarantee, which fields known consumers rely on, and where the boundary is with backends that do transform the request.

Passthrough Backends

The following backend types do not apply a backend-specific transform in their chat-completions transport path. Once a request has reached that path, all non-standard top-level fields, extra_body, and unrecognized keys are forwarded unchanged:

  • OpenAI (with the cloud-boundary exception described below)
  • Llamacpp (llama-server)
  • Mlxcel (reuses LlamaCppBackend; see src/infrastructure/backends/factory/backend_factory.rs)
  • Ollama
  • vLLM
  • SGLang (reuses VLLMBackend; see src/infrastructure/backends/factory/backend_factory.rs)
  • LMStudio
  • LocalAI

The passthrough has three equivalent proxy sites plus one factory-backed backend site:

  • HTTP path: src/proxy/backend.rs::make_http_request, the primary passthrough site. The payload.clone() call in the else branch (non-Anthropic, non-Gemini) forwards the body unchanged.
  • Unix-socket path: the sibling else branch in make_unix_socket_request within the same file, exercised when backend.transport is UnixSocket.
  • Streaming path: src/http/streaming/handler.rs around the client.post(&backend_url).json(&current_payload) call, which forwards current_payload verbatim for each attempt in the streaming fallback loop.
  • Factory-backed llama.cpp / MLxcel path: src/infrastructure/backends/llamacpp/backend.rs::LlamaCppBackend::execute_chat_completion, reached for BackendTypeConfig::Llamacpp and BackendTypeConfig::Mlxcel via src/infrastructure/backends/factory/backend_factory.rs.

Apart from the three boundary exceptions described next, none of these sites applies any provider-specific transformation.

Cloud OpenAI exception

The OpenAI backend type covers both the OpenAI cloud and local OpenAI-compatible engines, and the two disagree on unknown fields: api.openai.com rejects any unrecognized top-level key with HTTP 400, while local engines consume them. The transport sites therefore apply one gated filter. When a /v1/chat/completions request is bound for a URL containing api.openai.com, the engine-only fields chat_template_kwargs, thinking_budget_tokens, enable_thinking, preserve_thinking, top_k, min_p, and repeat_penalty (NON_OPENAI_FIELDS in src/infrastructure/backends/field_filter.rs) are removed before the request is sent.

  • The strip runs at every cloud-OpenAI chat send site: the non-streaming passthrough branch of make_http_request, the streaming transport, and each hop of the mid-stream and auto-selection fallback loops. A fallback may switch backends, so the gate is re-evaluated per hop.
  • The same denylist backs the cloud Gemini strip (Google's /v1beta/openai/chat/completions endpoint validates field names just as strictly). Azure OpenAI and the Anthropic /v1/messages bridge are separate surfaces and are not covered by this gate.
  • Stripping is top-level only. extra_body and reasoning_effort are never touched: extra_body is the intentional escape hatch for provider-specific settings, and reasoning_effort is accepted by cloud OpenAI directly (cloud Gemini maps it to thinking_level).
  • Local engine URLs never contain api.openai.com, so the filter is a no-op for llama.cpp, vLLM, SGLang, MLxcel, Ollama, LM Studio, and LocalAI, and the passthrough guarantee for them is unchanged.

Anthropic-native field exception

The client-facing OpenAI ingress accepts three Anthropic-native extension fields on an otherwise canonical chat-completions body: speed (fast mode) and the extended-thinking pair thinking and output_config (ANTHROPIC_NATIVE_FIELDS in src/infrastructure/backends/field_filter.rs). The router converts them on the Anthropic side of dispatch. No OpenAI-wire target reads them under these names: llama.cpp reads chat_template_kwargs and thinking_budget_tokens, vLLM and SGLang read chat_template_kwargs, and the canonical reasoning spelling every target understands is reasoning_effort. Cloud OpenAI, Azure OpenAI, and cloud Gemini answer HTTP 400 for an unrecognized top-level key, so leaking one fails the request outright rather than being ignored.

They are therefore removed from a /v1/chat/completions body whenever the selected backend's configured backend_type speaks an OpenAI wire, at every send site: the non-streaming passthrough branches of make_http_request and make_unix_socket_request, build_openai_chat_request_core in the streaming transport (which covers direct streaming, the mid-stream per-hop loop, and auto backend selection), stream_via_unix_socket, and GeminiBackend::transform_request.

  • The gate keys on the configured type, not on the URL and not on the model name. Unlike the cloud-OpenAI strip above, it covers local engines too, because no local engine reads these names either.
  • Generic, Openai, Azure, Gemini, Vllm, Ollama, Llamacpp, Mlxcel, Lmstudio, and Sglang strip. The match is exhaustive with no catch-all arm, so a new backend type forces an explicit decision.
  • Anthropic and Bedrock keep the fields: their transform is the code that reads thinking and output_config, and speed stays governed by the backend's separate anthropic_fast_mode opt-in.
  • Continuumrouter keeps them too. A downstream Continuum Router understands the same canonical extension fields and applies this same rule against its own backends, so stripping here would discard the client's reasoning intent on a hop toward a downstream router that fronts Anthropic.
  • Full passthrough happens only when the selected backend's name is absent from the configuration snapshot the send site consults, for example a hot reload that removed or renamed the backend while the request was in flight. A backend declared without type: in YAML is not this case: backend_type defaults to generic, so it strips like every other OpenAI-wire type.
  • Stripping is top-level only. extra_body and reasoning_effort are never touched, for the same reasons as the cloud-OpenAI strip.
  • The removal is preceded by a translation, in the same gate and on exactly the requests that strip: the thinking config is converted into a reasoning_effort value the selected target accepts, so the client's reasoning intent crosses the hop under the one spelling every OpenAI-wire target understands instead of being discarded. A client-supplied reasoning_effort (or reasoning.effort) always wins, and a malformed config degrades to a plain strip. The mapping table and the per-target vocabulary rule are documented in Model fallback.

OpenAI Chat Completions compatibility exception

After the actual backend and model have been selected, OpenAI and Azure OpenAI Chat Completions requests pass through one idempotent compatibility transform at the provider boundary. For the o1, o3, and delimited GPT-5 families (for example gpt-5, gpt-5-2025-08-07, and gpt-5.6-luna), it renames the legacy max_tokens field to max_completion_tokens. Matching is case-insensitive but family-boundary aware, so unrelated identifiers such as o10, gpt-50, and gpt-5o do not match.

The same transform is used by selected HTTP and Unix-socket proxy sends, streaming attempts and fallbacks, Anthropic Messages compatibility bridges, typed OpenAI chat and streaming calls, control-plane probes, and OpenAIBackend::transform_request. The public continuum_router::proxy::transform_payload_for_openai helper remains a thin wrapper over that source of truth.

The transform has these precedence and preservation rules:

  • If only max_tokens is present, its value moves to max_completion_tokens.
  • If both fields are present, the caller-provided max_completion_tokens value wins and max_tokens is removed.
  • Reapplying the transform is a no-op. Sampling controls such as temperature, top_p, and logprobs, plus extra_body and all unrelated fields, remain unchanged.
  • A model name alone never activates OpenAI rules. Gemini and self-hosted backends keep their original token field even when the selected model is named gpt-5.6-luna.

Operator-configured request extensions

A backend that configures request_extensions (see Request Extensions) receives one further, operator-authored change that is not a provider transformation: the configured body.defaults and body.overrides fragment is merged into the fully built wire payload after every strip and normalization above, and the configured headers are attached. The merge is one shared function in src/infrastructure/common/request_extensions.rs called at each send site with the selected backend's configuration, so it applies per attempt and per fallback hop. Configuration load refuses any body key the strips above remove for that backend, so the order can never re-add a stripped field. A backend without the block is passed through exactly as described here.

Known consumers

The following fields are used in production and must survive the passthrough unchanged:

  • chat_template_kwargs — Jinja template parameters for llama.cpp / MLxcel (e.g., {"enable_thinking": false, "preserve_thinking": true}). Required for Qwen3-family thinking-mode control.
  • thinking_budget_tokens — per-request thinking-budget cap accepted by llama.cpp. Required for Qwen3 thinking mode via llama-server.
  • extra_body — object of additional parameters passed by OpenAI client libraries (e.g., the Python openai SDK's extra_body kwarg). Used to forward vLLM-specific settings such as {"skip_special_tokens": false}.

Transform Backends

The following backend types run the request through a provider-specific transformation layer before forwarding it. Non-standard fields may not survive verbatim:

  • Anthropic — src/http/handlers/anthropic/transform.rs. Among other changes, the thinking block (with budget_tokens) is transformed into reasoning_effort for the downstream OpenAI-compatible call.
  • Gemini — src/infrastructure/backends/gemini/transform.rs. Gemini-native request reshaping; consult the transform source for the current field mapping.

Integration Tests

The transport-layer passthrough contract is protected by the following integration tests:

  • tests/llamacpp_passthrough_test.rs — verifies that chat_template_kwargs, thinking_budget_tokens, extra_body, and arbitrary unknown fields reach the llama-server endpoint unchanged.
  • tests/openai_request_compat_test.rs — verifies typed OpenAI chat, streaming, request transforms, and control-plane probes against a real captured wire request.
  • tests/mlxcel_passthrough_test.rs — keeps separate coverage for the Mlxcel backend type. It includes a proxy-path fixture and a factory-backed BackendFactory -> LlamaCppBackend::execute_chat_completion assertion that detects MLxcel wiring divergence as a distinct test failure.
  • tests/sglang_passthrough_test.rs and tests/sglang_streaming_passthrough_test.rs — keep separate coverage for the Sglang backend type: SGLang request extensions (top_k, min_p, lora_path, session_params, chat_template_kwargs, and others) reach the engine unchanged on both the non-streaming and streaming paths, reasoning_content is preserved on the response message and on stream deltas, and the final usage chunk is relayed when stream_options.include_usage is set.
  • tests/anthropic_input_test.rs — includes a negative test (test_anthropic_thinking_budget_tokens_transforms_to_reasoning_effort) that confirms thinking.budget_tokens is transformed and does not reach the downstream server as a raw thinking_budget_tokens field.
  • Unit tests in src/infrastructure/backends/field_filter.rs — cover the cloud-gated strip: a cloud OpenAI chat request loses every denylisted field, a local-engine request keeps them all, and non-chat endpoints are left alone.
  • The same unit-test module covers the Anthropic-native field gate, one case per branch: every OpenAI-wire type strips, Anthropic / Bedrock / Continuumrouter keep, an unknown type keeps, a non-chat endpoint keeps, and extra_body and reasoning_effort survive.
  • tests/cross_provider_fallback_payload_test.rs: asserts the Anthropic-native strip on the outbound body a mock backend actually received, including a custom-alias hop onto a vLLM-typed fallback (non-streaming and streaming), an Anthropic-typed target behind a custom alias that must still receive thinking, and an Anthropic to Anthropic hop where speed reaches the wire only under the anthropic_fast_mode opt-in. The same suite covers the conversion that precedes the strip: an adaptive thinking config with an effort, the budget bands, max toward targets with and without xhigh, a target that supports no reasoning_effort at all, an existing reasoning_effort winning, disabled thinking, a malformed config, and the streaming send path.

If a refactor breaks any of these tests, treat it as a breaking change to a public contract, not a routine test failure. Update this document to reflect the new behavior before merging.

What the Guarantee Covers

The guarantee applies to the transport sites listed above. It does not mean the upstream server always receives a byte-for-byte copy of the client-submitted JSON body.

Before those passthrough sites run, chat_completions can still perform router-level preprocessing:

  • Global system prompt injection when global_prompts is configured, which modifies the messages array.
  • Router-managed web_search injection for eligible self-hosted backends when web_search.enabled applies, which can add a tools entry before the upstream call.

At the selected provider boundary, only OpenAI and Azure OpenAI chat requests receive the token-field normalization described above. Non-OpenAI backends are not rewritten based on the model string. No additional top-level JSON keys are injected, and no passthrough fields are filtered or renamed except for that normalization, the cloud OpenAI strip, and the Anthropic-native field strip that every OpenAI-wire backend_type applies.