Skip to content

Middle-Out Message Transform

This page records the automatic context fitting that runs on POST /v1/chat/completions and matches OpenRouter's middle-out transform (issue #2415).

Activation

The transform needs both an operator switch and a client opt-in. Without both, the request is forwarded unchanged and an oversized input keeps the backend's context-length error.

  • Operator switch: proxy.middle_out.enabled, default false. The block is read from the request's configuration snapshot, so a reload applies to the next request.
  • Client opt-in: a top-level "transforms": ["middle-out"] array in the request body. The match is ASCII case-insensitive, and a transforms value that is not an array is ignored.

When the transform drops messages, it removes the spent middle-out entries from transforms and removes the key when the array becomes empty, so no backend sees a directive the router already honored. Other transform names survive. When the transform does not run, the payload is forwarded byte-identical.

Fit rule

The target is the model's effective context window minus the output budget:

  • The window comes from the same cached limit snapshot as the primary context-window admission check, resolved per serving backend with the request's candidate allow-list. No backend I/O is involved.
  • The output budget is a non-null max_completion_tokens when present, else a non-null max_tokens, else proxy.middle_out.default_output_reserve (default 1024, range 0..=1000000). A reserve of 0 is legal and means no reserve.
  • The input estimate is the shared typical estimate in src/core/prompt_estimate.rs, the same figure fallback budget planning and smart routing read.

The transform runs only when the estimate exceeds the target. It skips a request when the context window or the target is unknown or zero.

Drop algorithm

  • The leading run of system or developer messages is pinned and never dropped.
  • An assistant message with a non-empty tool_calls array (or a legacy non-null function_call) forms one unit with the contiguous tool or legacy function results that follow it. Every other message is its own unit, and a unit is never split.
  • The unit holding the last user message and every later unit are protected, so the current turn survives whole. With no user message, the last unit is protected.
  • Whole units are dropped from the front of the non-pinned region, oldest first, until the estimate fits. Because the drop set is a contiguous prefix, the oldest middle history goes first and the most recent turns stay.

If dropping every unprotected unit still exceeds the target, the transform does not fit the request. The payload is left untouched and the request proceeds as it would without the transform: the context-window fallback when configured, otherwise the backend's context-length error.

Response signal

A fitted response carries the x-continuum-middle-out header with dropped=<n>;estimated_input_tokens=<n>;target_input_tokens=<n>;context_window=<n>, on buffered, streaming, and early-refusal responses alike. The response body keeps its normal shape. The router also emits an info log event with the routed model, dropped count, post-fit estimate, target, and context window.

Fallback and metering

  • Fitting runs before dispatch on the same limit snapshot as admission, so a fitted request passes admission and does not also take a context-window fallback hop (issue #2307).
  • The fitted payload is what the backend receives, and usage settlement reconciles against the usage the backend reports, so the completed request is charged for the fitted input.
  • The token-budget admission hold is taken from the original client body in the rate-limit middleware, before the transform runs, so an over-window request reserves its pre-fit estimate until settlement releases the difference.

Scope

The transform covers POST /v1/chat/completions only. The Responses bridge behind this route inherits the fitted payload; every other ingress forwards unchanged.