Changelog¶

All notable changes to Continuum Router are documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

v2.0.0 - 2026-11-03¶

Added¶

• Added observability.callbacks (#2376), an opt-in first-party export of one trace per request to an LLM observability platform's native schema. Langfuse ships, over its OTLP/HTTP endpoint, reusing the prompt-log export mapper; Braintrust, Arize, and Langsmith are follow-ups. The section is off by default and independent of tracing.enabled. Prompt, completion, and tool-call text is exported only when include_content: true and prompt-log capture admitted the request after its redaction pass; credential values never leave the environment and are used only for HTTP Basic auth. Delivery is fail-open through a bounded queue with a dedicated worker and bounded retry per sink, so a slow platform never delays or fails a response; observability_callback_events_total{sink,result} and observability_callback_queue_depth report delivery. The configuration assistant, the generated IDE rule files, the design record (English and Korean), the example config, a template, and the admin config surface were updated.

• Added the optional notifications: section (#2416), which pushes client-facing rate-limit and quota events to application webhooks. With enabled: true, the router POSTs a signed, bounded JSON event for rate_limit_exceeded (local request-rate, token-budget, in-flight, and capacity refusals, plus Hub tier minute limits), quota_low (the first crossing of a Hub monthly budget fraction), and quota_exhausted (Hub monthly budgets, Hub cost-center and model budgets, local scoped budgets, and the legacy per-key max_tokens_budget). Producers only gate, coalesce, and enqueue, so a slow or failing receiver never delays or fails a request; each target owns a bounded queue and one delivery task, with retries bounded by retry.max_attempts and backoff capped at retry.max_delay. The payload carries the key id, static event and limit names, and bounded numbers, never key material, prompt or completion text, or request headers. With signing_secret, every request also carries X-Continuum-Timestamp and an X-Continuum-Signature HMAC over the exact sent body, and the secret is masked in configuration output and supports ${ENV_VAR}. Endpoints are validated at load and again at apply with the strict SSRF validator, resolved addresses are pinned, redirects are disabled, and a private receiver needs allow_private_network: true. New metrics: notification_deliveries_total, notification_retries_total, notification_queue_depth, notification_dropped_total, and notification_coalesced_total. The example config, the configuration assistant, the generated IDE rule files, and the English and Korean webhook notification pages were updated.

• Added opt-in model variant suffixes on /v1/chat/completions (#2405). With proxy.variant_suffixes.enabled: true, a model name that does not resolve exactly may carry one recognized trailing suffix: :nitro prefers the highest fresh measured generation throughput among healthy admissible candidates, :floor prefers the cheapest estimated cost using the same pricing and cost estimate as least_cost, :thinking raises an unset reasoning_effort to high on a model whose metadata declares thinking support, and :online runs the router-managed web_search loop. The full requested name is resolved first, so a served id containing a colon (an Ollama name:tag) and an unknown suffix word keep their existing behavior, and the grammar is not interpreted on any other ingress. :nitro and :floor never refuse: with no ranking data the normal strategy serves the request and the response carries x-continuum-model-variant-applied: false; :thinking without declaring metadata and :online without a usable executor (including a streaming request) answer 404 unsupported_model_variant. Per-key allowed_models, the per-model rate limit, and the per-model token budget key on the base id, which is dispatched upstream, while the requested name is echoed in the response model, x-continuum-model-variant names the variant word, the request is counted in continuum_model_variant_requests_total{variant, applied}, and a fallback-chain hop never inherits the variant. The configuration assistant, the example config, and the English and Korean configuration and design-record pages were updated.

• Added optional inbound JWT/OIDC bearer validation under api_keys.jwt, off by default and behind the jwt-auth Cargo feature (outside default/full) (#2375). A static key always wins; only a key miss is examined as a JWT, and a token whose unverified iss is not the configured issuer falls through to the old key-miss path. A token naming the issuer is verified against an https or loopback JWKS (or inline static_jwks) and must pass iss, aud, exp, and nbf with bounded clock skew, using only configured asymmetric algorithms; alg: none, HMAC, kty: oct, and token-supplied jku/jwk/x5u are refused. The JWKS fetch has a hard timeout, follows no redirects, and streams under a size limit, and the cache uses a TTL plus a rate-limited single-flight refresh on an unknown kid. A valid token produces the usual AuthContext with configurable subject, user, team, scope, model, and backend claim mappings; a JWT scope list is authoritative, so an empty mapped list grants no scopes. Rejections and JWKS outcomes are counted by continuum_jwt_auth_total{result} and continuum_jwt_jwks_fetch_total{result}. A build without the feature validates the section but warns that it is inert. If an enabled section cannot be built into a validator, static keys keep working and a token naming the configured issuer is refused and counted misconfigured. The admin config schema, the configuration assistant, the generated IDE rules, the example config and a template, and the English and Korean security and design pages were updated.
• Added a /v1/vector_stores passthrough (#2412), so OpenAI SDK clients can create, list, retrieve, modify, and delete vector stores, attach, list, retrieve, and remove files, and search a store through the router while keeping its auth, API-key scopes, rate limiting, and usage attribution. A store call names no model, so create and list select one vector-store capable backend with the configured strategy, and the router records (principal, store id) to the backend that created the store; every store-scoped call re-checks that binding against the live configuration and the caller's allowed_backends, and an unknown or foreign id answers 404 vector_store_not_found without contacting the backend. A list keeps only the caller's own stores, and JSON request bodies are bounded at 1 MiB. Only openai and generic types on http/https are candidates, and a request no candidate can serve answers 400 vector_stores_not_supported rather than a router 404. The routes do not run request-parameter policy, guardrails, response caching, or the per-model generation budget, and each call is metered as a non-completion request with zero tokens. The English and Korean API pages and a design record were added.
• Added the middle-out message transform for /v1/chat/completions (#2415). With proxy.middle_out.enabled set and a client "transforms": ["middle-out"] opt-in, an input whose shared typical estimate exceeds the target model's context window minus its output budget is fitted by dropping whole messages from the middle while keeping the leading system or developer prompt and the current turn; a tool call and its tool results are dropped together. A fitted response carries x-continuum-middle-out with the dropped count, post-fit estimate, target, and context window, and the spent transform entry is removed before dispatch. A request that does not opt in, a disabled transform, an input that already fits, or one the kept messages alone cannot fit forward byte-identically, and a fitted request does not also take a context-window fallback hop. The configuration assistant, the generated IDE rule files, and the English and Korean middle-out design pages were updated.
• Added per-key data-residency routing (#2374). A backend can carry an operator-defined region_name tag, and an API key with a non-empty allowed_regions list may only route to backends carrying one of those tags, matched exactly and case-sensitively. A backend without a tag never qualifies, the list intersects with allowed_backends, and an empty intersection fails closed with 403 instead of routing outside the requested region. The effective list is applied on every dispatch path, fallback hops and hedges included, and keys with different region scopes never share a response-cache entry. A tag no configured backend declares produces a startup warning. The configuration assistant, a config template, the example config, and the English and Korean configuration pages were updated.
• Added circuit_breaker.latency_as_failure (#2964), so a backend that answers 2xx but takes longer than ttft_ms (streaming, provider dispatch to first byte) or total_ms (buffered, whole response) counts that dispatch as one terminal failure. The backend leaves rotation at the existing failure threshold and returns through the ordinary half-open probe, which a slow probe reopens; a fast result is an ordinary success, never a success followed by a failure, and the two thresholds never cross-apply. minimum_samples (default 10) is a warm-up in judged observations since the circuit last closed or had its window cleared, and half-open probes skip it; model_overrides gives a served model id its own thresholds and floor. The buffered chat path judges the completed response, and the standard chat streaming path defers the handshake outcome to the relay's first-byte stamp, while the other streaming relays still settle at the handshake. Transitions carry reason="slow" on circuit_breaker_transitions_total and slow failures are counted as circuit_breaker_failures_total{error_type="slow"}. The configuration assistant, the IDE rule files, three config templates, the example and sample configs, the metrics and load-balancing pages, and the English and Korean circuit-breaker pages were updated.
• Added retry.prefer_untried_backends (#2965), off by default: when enabled, a retry selects among the eligible backends the same request has not dialed yet, and falls back to the full eligible set when every eligible backend has been dialed, so a single-backend deployment still gets every attempt. The dialed set lives on the retry state, shared by every attempt clone, and is recorded at the buffered dispatch seam after selection, the hop dial bound, and circuit/rate-limit admission succeed, so a backend that was selected but refused at admission is not counted. false keeps the prior behavior, where a retry may re-select the backend that just failed; replay: disabled still governs side-effecting backends. The configuration assistant, the example config, the admin config schema, and the English and Korean configuration pages were updated.
• Added logging.access.completion_line, success_level, client_error_level, and server_error_level (#2402). The request logging middleware now writes one completion line per request that reaches a handler, at the level its status selects: 2xx and 3xx use success_level (default info), 4xx use client_error_level (default warn), and 5xx use server_error_level (default error), each accepting debug, info, warn, or error. The line also carries the request id, user id, organization id, and client address as structured fields. The line is off by default, so a configuration that omits logging.access keeps the existing output, an unknown level spelling fails the load, and an excluded path writes no completion line. A request refused before a handler stays with event=request_rejected. Request and response body capture remains out of scope under the opt-in prompt-logging epic #2294, and log_request_body or log_response_body is a load error. The configuration assistant, the generated IDE rule files, the example config, the admin config schema, and the English and Korean logging pages were updated.
• Added streaming.mid_stream_fallback.same_model_first (#2966), off by default: after a mid-stream failure or stall, the committed OpenAI-wire relay first tries another backend that serves the same public model, and only then advances fallback.fallback_chains. The sibling set is user-routable, permitted for the API key, servable by the relay, admitted by its circuit breaker, and excludes backends already dialed for that model in the request; the hop keeps the client's model, reuses the relay's continuation or restart prompt, and counts toward max_fallback_attempts. A sibling that also fails, an open sibling circuit, or an empty set falls through to the chain unchanged. The superseded attempt records fallback_source and the sibling is a normal successor under the same request_id, with streaming_fallback_total{reason="same_model_sibling"}; mid-stream recoveries still emit no X-Fallback-* headers because the response is already committed. The example config, the configuration assistant, and the English and Korean architecture and configuration pages were updated.
• Added backends[].model_configs[].upstream_id (#2963), so one public model ID can pool backends that serve it under different names. The configured id stays the routing identity everywhere the router or the client sees it (selection, per-key allowed_models and request_params scopes, /v1/models, fallback-chain lookup, and the usage ledger), while upstream_id is the model the router puts on the wire for that backend, and the models/{id} path segment on native Gemini calls. Each attempt is rewritten for the backend that receives it, so retry and fallback hops carry their own spelling, and the response model is restored to the public ID in buffered JSON bodies and in OpenAI-wire SSE chunks. A backend's live catalog contributes the public ID, so the pool lists one entry, and the upstream spelling is not routable unless a backend publishes it literally. config validate rejects a blank upstream_id and a value that collides with another model_configs[].id on the same backend; the same public ID may map to different upstream IDs on different backends. The English and Korean advanced-configuration pages were updated.
• Refused requests now produce a refused usage record (#2368). A request the router declines before dispatch carries one record with the reason code, the requested model and key when they resolved, the client-visible status, and zero token counts, so the Hub can show the request that no backend ever saw. Only routing, entitlement, and per-backend rate-limit refusals qualify; the record uses the router provider sentinel and carries no attempt identity, and the Hub keeps it out of every usage rollup. The vendored continuum-protocol moved to Hub rev 09f45d3c, which learned the outcome and its refusal_reason field. New metric label value usage_records_total{outcome="refused"}; the English and Korean architecture pages were updated.
• Added fallback.empty_reply_retry (#2440), off by default: a non-streaming /v1/chat/completions reply whose first choice has no message, or whose message.content (else message.refusal) is missing or whitespace only, advances the existing fallback chain. The hop reuses the fallback.quality_check hop path, so it continues the chain's attempt numbering and counts against max_fallback_attempts, the hop timeout multiplier, the fallback dial permit, both key allow-lists, and the non-streaming chain budget, while max_extra_hops (1-5, default 1) additionally caps this trigger's own hops. A tool_calls reply and a reply with finish_reason: length never trigger, and a request fallback.quality_check covers skips the trigger. The trigger inspects only the body the guardrail output gate passed, a hop's reply passes the same gate, and a failed, over-budget, or blocked hop serves the latest gate-passed reply. Scope matches proxy.internal_streaming: a buffered request with one choice, no log probabilities, a text or json_object response format, and an OpenAI-wire backend over HTTP(S), while unix:, native azure/anthropic/gemini/bedrock, and replay: disabled keep their existing behavior. New metric fallback_empty_reply_retry_total{result}. The configuration assistant, the IDE rule files, a config template, the example config, the admin config schema, and the English and Korean architecture, configuration, and metrics pages were updated.
• Added opt-in shadow traffic mirroring (#2468). The shadow section replays a sampled share of matching client requests to a second model after the primary request is dispatched, on a detached task, so the primary response, its headers, and its client-visible latency are unchanged and a shadow failure is recorded and never surfaces. The mirror charges no client rate-limit quota because it never re-enters the HTTP layer, charges no per-model or per-backend bucket, moves no circuit state, and writes no usage row. Client credential headers (authorization, x-api-key, api-key, x-goog-api-key, cookie, set-cookie, proxy-authorization, x-auth-token, x-access-token), router-owned attribution headers, client identity and tenant headers (x-user-id, x-tenant, x-org-id), router-control and forwarding headers, and framing headers are stripped, and the shadow backend authenticates with its own configured credential. Sampling is deterministic from a stable hash of the API key id or the user id (sample_key: api_key or user) and random per request for none; a body over max_request_bytes is skipped and counted, max_concurrent sheds rather than queues, timeout bounds one whole attempt, and a streamed shadow response is drained without accumulating. New metrics are router_shadow_requests_total{result}, router_shadow_latency_seconds{model,backend}, and router_shadow_usage_tokens_total{model,backend,kind}. Prompt and completion content is never logged. An absent or disabled section leaves request handling unchanged. The configuration assistant, the IDE rule files, a config template, the example config, and the English and Korean shadow-traffic pages were updated.
• Added per-response cost and token trailers for streaming responses (#2546). proxy.response_cost_headers, which already covered non-streaming responses, now also puts x-continuum-cost-microusd, x-continuum-tokens-in, x-continuum-tokens-out, x-continuum-tokens-cached, and x-continuum-latency-ms on the metered streaming pipelines (chat completions, completions, the Responses API, and their bridges) as HTTP trailers: the head announces them in a Trailer header, and the trailer section after the last frame carries the values from the same pricing arithmetic and the same omissions as the headers. The router's own finalized stream usage is the source, so no backend usage-trailer reading is involved. On HTTP/1.1 a client asks for the fields with TE: trailers (hyper encodes them only then), and a client that ignores trailers still reads the complete body; HTTP/2 carries them without negotiation. HTTP/1.1 and HTTP/2 cleartext are both covered, and the configuration assistant and the English and Korean configuration pages were updated.
• Added the json_schema guardrail provider, which validates the assistant's structured output against the JSON Schema the client declared on that request (#2470). It reads response_format.json_schema.schema (OpenAI Chat Completions), response_format.schema (llama.cpp), text.format.schema (OpenAI Responses), and output_config.format.schema or output_format.schema (Anthropic Messages), so nothing is configured statically and a request that declares no schema is a no-op. The dependency-free validator covers type, enum, const, required, properties, additionalProperties: false, items, minimum, and maximum, ignores $ref and pattern so a client schema cannot make the router fetch a URL or compile a costly regular expression, and is bounded by depth and violation count. monitor counts violations and passes the response through; enforce returns a schema_violation block whose reason names bounded JSON Pointers and the failing keyword, rendered through the configured block_behavior. options.repair: {enabled, max_retries (0-5), instruction} re-dispatches a buffered reply with the validation errors appended as a system instruction and serves the first reply that validates, while streamed replies are validated as an assembled message and never repaired. New metrics are router_structured_output_checks_total{result}, router_structured_output_violations_total{keyword}, and router_structured_output_repairs_total{result}, all with bounded labels. The configuration assistant, the IDE rule files, and the English and Korean guardrails pages were updated.
• Added backends[].responses_drop, a per-backend opt-in that lets a /v1/responses conversion backend drop the advisory native-only fields include (only when every requested value is allowlisted, today just reasoning.encrypted_content), prompt_cache_key, safety_identifier, and reasoning.summary instead of refusing the request (#2867). The default stays the fail-closed refusal: with no list configured, every response and message is unchanged. Semantic fields (text, conversation, prompt_cache_options, prompt_cache_retention, multi_agent, context_management, max_tool_calls, reasoning.mode, reasoning.context, tools, programmatic calling, and native input items) always refuse, an unknown or semantic name is a configuration load error, and responses_api: native ignores the list. The reply discloses the removed fields in a sorted, deduplicated x-continuum-dropped-fields header, exposed through CORS and replayed on a response-cache hit, and responses_drop_fields_total{field} counts one increment per dropped field per served request. Dropping prompt_cache_key leaves the router's prompt, prefix, and semantic cache keys unchanged, and an opted-in Codex CLI request stays response-cache eligible. The Codex CLI guide, the backend configuration docs, the configuration assistant, and the example and sample configs were updated in English and Korean.
• Added usage-weighted backend selection (#2452). routing.usage_weight adds an additive scorer term that ranks candidates by the share of their declared backends[].provider_tpm / provider_rpm budget the router has not yet spent, from a router-local per-backend consumption window fed by the existing token metering, so it works for cloud backends with no new data source. A declared candidate scores weight x remaining_share; a candidate that declares nothing, or has spent its whole budget, scores exactly 0.0 and keeps its base score, so an unknown quota is never treated as infinite or as zero. The term composes with the KV overlap and engine-load terms; it decides a pass when it strictly prefers one candidate and the non-usage terms would not have selected the same backend, so a genuine KV hit keeps its KV metrics. A tie falls back to selection_strategy. provider_tpm and provider_rpm are declarations and enforce nothing. The section is off by default and selection is byte-identical when it is unset. Every pass the term decides or cannot settle records routing_usage_weight_decisions_total{backend,reason} with reason in applied, no_declaration, no_differentiation. The configuration assistant, the IDE rule files, the example config, and the English and Korean load-balancing, advanced configuration, and metrics pages were updated.
• Added a Codex CLI integration guide, docs/en/guides/codex-cli.md and its Korean pair (#2851). It separates the CLI as a router client from the ChatGPT-subscription upstream, shows ~/.codex/config.toml for the responses and chat wires, states that openai and azure pass /v1/responses through by default while other OpenAI-wire types need responses_api: native, and maps the observed 400 unsupported_request_parameter refusals to their cause and remedy. A new tests/responses/codex_cli_test.rs posts a captured codex-cli 0.166.0 body at a native backend and asserts the forwarded include, prompt_cache_key, store, and tool list, then at a conversion backend and asserts the refusal names prompt_cache_key. The native-only refusal now names both ways forward: set responses_api: native on a backend that serves the Responses API itself, or send the request over the Chat Completions wire from a client that can omit the field.
• Added the data-plane budgets: section for long-window token and cost quotas (#2461). budgets enforces rolling-window limits per API key, per model, and globally, independent of the control plane: windows names the rolling periods, scopes sets tokens and cost_usd per scope, storage selects per-process memory or shared Redis, and fail_open decides whether a storage outage admits or refuses. It extends the #2323 tracker rather than adding a second one: api_keys[].max_tokens_budget stays the explicit per-key override and the section's per_api_key limit is the fleet default for keys without one, so a request is never charged twice for the same scope. Cost comes from model-metadata.yaml pricing in integer micro-USD, and an unpriced model counts tokens but no cost. A refusal is a 429 with type: budget_exceeded naming the scope, window, and configured limit, in the ingress protocol's error envelope; remaining is reported only for the identity-bound per-key scope and withheld for shared scopes, and X-Budget-* headers carry the per-key level. Completed usage is recorded exactly once from the usage seam for buffered and streamed responses on both the OpenAI and Anthropic wires. GET /admin/budgets reports current counters and POST /admin/budgets/reset clears a scope's window, with router_budget_rejections_total{scope,window} and router_budget_used{scope,window,unit} metrics. The section is hot-reloadable and absent or enabled: false is a strict no-op. The configuration assistant, the IDE rule files, two config templates, the example config, and the English and Korean configuration, security, and design-record pages were updated.
• Added opt-in fair queuing across tenants (#2397). rate_limiting.fair_queuing holds a request an identity dimension (per_api_key, per_user, per_team, per_model) refuses in its tenant's bounded queue instead of answering with a bare 429, and serves the queues by weighted fair scheduling so one tenant's flood cannot starve another. The tenant is the same user, key, or team identity the limiter already charges (tenant_dimension, default user) and weights sets each tenant's share. Every level is bounded (max_queue_depth_per_tenant, max_total_queued, max_tenants, max_in_flight, max_wait_ms), queued time counts against the request's total timeout, and a queued request is re-checked against the limiter when its turn comes. A request refused at or after the queue carries x-fair-queue-position and x-fair-queue-wait-ms, an admitted request that waited carries those plus x-fair-queue-estimated-wait-ms, and error.details.fair_queue names the tenant dimension, reason, position, and wait. Global and per-client refusals stay immediate because their mounts run before authentication, and deferred_admission still handles token-budget refusals, so both modes can be enabled together. New metrics are rate_limit_fair_queue_depth, rate_limit_fair_queue_tenants, rate_limit_fair_queue_wait_seconds{tenant_dimension}, rate_limit_fair_queue_grants_total{tenant_dimension}, and rate_limit_fair_queue_refused_total{tenant_dimension, reason}. The configuration assistant, the IDE rule files, two config templates, the example config, the admin schema, and the English and Korean rate-limiting, architecture, and design-record pages were updated.
• Added rate_limiting.limits.*.max_in_flight_requests, a per-tenant concurrent in-flight cap on per_api_key, per_user, per_team, and per_client (#2383). A request over its tenant's cap is refused immediately with 429, limit_type: "inflight", error.details.inflight_dimension naming the dimension, and Retry-After: 1, and it is never queued. The slot is taken after every request-count and token-budget check passes, so a refused request occupies none, and it is released by a guard attached to the response body, so a stream holds it until its last frame and a handler failure or a client disconnect frees it. The cap is off by default (0 = unlimited), ignored on global, per_backend, and per_model (config validate warns), accounted per replica, and exempted by the new inflight scoped-bypass class. New metrics are rate_limit_inflight_requests{dimension,tenant} and rate_limit_inflight_refusals_total{dimension,tenant}. The configuration assistant, the IDE rule files, three config templates, the example config, the design record, and the English and Korean rate-limiting, architecture, and changelog pages were updated.
• Added rate_limiting.tier_overrides (#2406), a map keyed by Hub tier_id that gives a tier its own per-key request rate, burst, and minute token budget. The limiter resolves the tier from the Hub-synced key table on every request, the same identity smart_routing.routing_policies[].when.key_tier names, and an entry changes the effective rate and burst for keys of that tier, plus tokens_per_minute when the entry writes one. An explicit api_keys[].rate_limit (with its burst and minute-budget overrides) remains the most specific setting and wins over the tier, and the tier wins over limits.per_api_key. The resolved tier is part of the request-rate and minute-budget bucket keys, so a key moved between tiers starts on the new tier's bucket, and the x-ratelimit-* headers report the limits actually applied. Without the control-plane feature, an enabled control plane with policy enforcement, or a synced policy, the map is inert and every key keeps the settings it would have without it (config validate and the startup log warn). These are local limits: the Hub tier's rpm, input_tpm, and output_tpm windows (with the shared counters configured under control_plane.policy.minute_limits) are enforced separately and are neither replaced nor widened, so a request is admitted only when both admit it. config validate rejects a zero rate or burst, config_report rejects a burst below the rate, and an unknown field inside an entry is a load error. The configuration assistant, two config templates, the example config, and the English and Korean rate-limiting and architecture pages were updated.
• Added scoped per-key rate-limit bypasses and per-key burst and minute token-budget overrides (#2426). rate_limiting.scoped_bypass_keys exempts a key from only the classes it names (request_rate, tokens_per_minute, tokens_per_day) while bypass_keys keeps its all-or-nothing meaning. api_keys[].rate_limit_burst sets the bucket capacity of the request rate in effect for that key, and api_keys[].rate_limit_tokens_per_minute replaces the shared per_api_key minute token budget for that key (0 disables it); both are settable in the config file and through POST /admin/api-keys and PUT /admin/api-keys/{id}. A per-key limit change made through the Admin API now emits its own api_key_rate_limit_change audit line with the key id and the old and new values, without requiring admin.audit.include_body. Per-key values are excluded from configuration exports and the WebUI diff. A budget shared across one user's keys remains issue #2381. The configuration assistant, two config templates, the example config, and the English and Korean rate-limiting and Admin API pages were updated.
• Added the rate_limiting.limits.per_user and per_team dimensions (#2381), keyed on the identity an API key already resolves: the key's user id for per_user and its organization/team id for per_team. Several keys of one user, or every member of one team, now draw on one bucket instead of each key getting its own allowance. Both dimensions take the same settings shape as the others, including tokens_per_minute and tokens_per_day, which are shared the same way. A request whose key carries no resolved id (anonymous, a static key without one, or a Hub key with no synced team_id, whose continuum-hub placeholder is not a team identity) skips the dimension and is counted on rate_limit_skipped_total{dimension="user"|"team"}, never charged to a bucket shared with other identity-less keys. The buckets are keyed on a replica-stable hash of the identity in memory and in the Redis key space (per_user:<hash>, per_team:<hash>), and a config reload applies new settings immediately and resets the tenant buckets. A refusal names the dimension in error.details.limit_type (user or team) and is counted on rate_limit_exceeded_total{limit_type}; both dimensions are shared by more than one credential, so their levels are withheld from response headers. config validate and the startup log warn when a tenant dimension is configured but the build and configuration resolve no API-key identity. The configuration assistant, the IDE rule files, two config templates, the example config, the admin schema, and the English and Korean rate-limiting and architecture pages were updated.
• Added responses_tools.deny_resource_fields (#2891), a closed list of client-supplied upstream resource references to refuse with 400 unsupported_request_parameter naming the field. The default lists vector_store_ids and container_ids, which apply on every backend and to nested tools. previous_response_id, conversation, prompt, and item_reference are opt-in and apply on the pass-through strategy only.
• Added the responses_tools configuration section (#2873), so an operator decides which hosted tool types POST /v1/responses may forward. unmodeled_types (default deny) covers types such as mcp and image_generation, modeled_hosted_types (default allow) covers web_search, file_search, code_interpreter, computer_use, and programmatic_tool_calling, and allow entries (exact) and deny entries (any ASCII case) override both. function, custom, local_shell, tool_search with execution: "client", and namespace as a container are always allowed, and namespace members are judged at any depth. A refusal is 400 unsupported_request_parameter naming the tool, runs before backend selection and any cache lookup on every backend type, and an edit applies to the next request.
• Added the prompt_logging.export.hub_push runtime (#2764). In a control-plane build with control_plane.policy.enabled: true, an enrolled router runs one more prompt-log export task that pushes redacted records to POST /api/agent/v1/prompt-logs with the enrollment credential, on its own cursor starting at the end of the record files, every usage_push_interval_secs, at most the Hub's prompt_log_batch_max_records records (default 100, never above 500) and 16 MiB of encoded JSON per request, each side cut to 64 KiB. A record leaves only when its key is in the Hub's own key table with allowed, it was captured on that allowed rather than on default_consent, and no local denied outranks it; keyless records, records captured on the deployment default, local-only keys, and keys stating nothing never leave, counted as consent_keyless, consent_deployment_default, consent_key_absent, and consent_withdrawn under sink="hub". A Hub 404 drops the backlog and moves the cursor to the end of the files, 413 drops the request whole, 400, 403, and 409 are re-sent one record at a time so only the refused record counts, and 401, 408, 429, 5xx, and transport failures retry the same records with identical idempotency keys after a backoff capped at 60 seconds; records the Hub reports as already stored count as duplicate. A 404 discards the backlog for good, a router the Hub has retired has each record refused one by one, and a team's denied reaches the router only through the Hub's key table. The agent's HTTP client no longer follows redirects on any endpoint. The vendored protocol crate moves to hub 1e3dac51 (the PromptLogBatch and PromptLogAck wire types and RouterSettings.prompt_log_batch_max_records), the Hub key table's consent is indexed by key id when a policy is applied, make hub-compat gains a prompt-log stage, and the inert-setting warning now applies only to a build without the feature or a deployment without control_plane.
• Added rate_limiting.limits.*.tokens_per_day, a daily token budget beside tokens_per_minute on every admission-time dimension (#2327), completing the epic #2401 token-budget surface. A text-generation request reserves its input estimate plus output limit against the day bucket of each configured dimension at admission and reconciles it against the reported usage like the minute bucket, and a request must fit both budgets of every configured dimension. The day bucket's capacity is one day of the rate and it refills at tokens_per_day / 86400 tokens per second as a leaky-bucket window, not a calendar-day reset at midnight; it composes with, and is distinct from, the max_tokens_budget rolling-window quotas of #2323 at the auth seam. A refusal names the dimension in error.details.token_dimension and the exhausted budget in error.details.token_budget (minute or day), and the identity-scoped per_api_key day budget reports on new x-ratelimit-tokens-day-{limit,remaining,reset} headers. The day bucket is Redis-backed under the token_budget_day key namespace with a TTL raised to at least one day. New metrics are rate_limit_token_budget_period_tokens_total{dimension,budget,outcome} and rate_limit_token_budget_period_refusals_total{dimension,budget}. per_backend.tokens_per_day is ignored and config validate warns about it. The configuration assistant, the IDE rule files, two config templates, the example config, and the English and Korean rate-limiting, architecture, and design-record pages were updated.
• Made the token-budget refusal's Retry-After size-aware and added the opt-in rate_limiting.deferred_admission mode (#2423). A refusal's Retry-After now reports the time the exhausted budget needs to refill the refused request's own cost rather than a fixed window: an in-burst denial keeps its deficit-over-refill wait, and an over-burst refusal (a request larger than the whole burst, which waiting can never admit) is computed from the request's cost against the budget's tokens_per_minute, floored at 1 second and capped at 86400. With deferred_admission.enabled: true, a refused request whose wait fits max_wait_ms (default 2000) is held for that wait instead of refused with a 429. The hold is bounded per router process by max_held_requests (default 1000) on a store count each replica keeps for itself, a held request holds no backend slot and no token reservation until a wait succeeds, the request-count dimensions are not re-charged on the retry, a client cancellation releases the hold through the guard's Drop, and a request larger than the whole burst is never held. A request past either bound gets the immediate 429 with the size-aware hint. The configuration assistant, the IDE rule files, the example config, the config report warning, and the English and Korean rate-limiting pages were updated.
• Added operator recovery for prompt log export (#2820). on_permanent_failure: skip on a sink gives up on a batch refused with status 400, 409, 413, 415, or 422, counting its undelivered records as prompt_log_export_total{result="dropped"}; the default block keeps retrying. cursor_id names a sink's cursor file instead of a hash of its destination, so a rotated webhook URL token keeps the cursor and the backlog goes to the new destination.
• Added prompt_logging_consent on locally configured API keys (#2819). A key states allowed or denied in api_keys[], an external keys file, the Admin API, or the WebUI, so a deployment with no Hub can run opt-in capture under default_consent: denied or exclude one key under default_consent: allowed. A denied from a local or Hub key wins, then a local allowed, then a Hub allowed, then default_consent, and the fail-closed Hub window still outranks a local value. A key without the field behaves as before. A disabled or expired key that states denied stays uncaptured in permissive mode, so disabling a key never widens consent; its allowed opts nothing in. A YAML startup or reload error from a wrong value type now names the field path.
• Added a config validate warning for a match-all backends[].models entry (* or models/*) when another backend is enabled (#2792). Such an entry does not keep a metadata alias on its backend, so the request can go to a remote provider that serves the canonical ID.
• Settled streaming usage against the token budget at the stream's end (#2420), completing the reservation lifecycle from #2395. A streamed completion that reaches its normal end settles the admission reservation against the finalized usage: the backend's usage block, or the input estimate plus the router's delta count when the backend reported none, so a normal end with no usage block still ends the reservation deliberately. A stream cut before its end (a client cancellation, a timeout, an upstream error, or a guardrail block) never settles; its reservation is released in full by the response body guard, and a truncated stream is never charged as if the stream completed. A mid-stream fallback hop never settles either; the successor attempt settles the reservation with the final reported usage. Settlement is exactly once, idempotent against the body guard's later release, and never blocks or delays the client response. The chat completions SSE pipeline, the Responses conversion and passthrough relays, the unix-socket relays, the responses-only bridge, and the Anthropic ingress stream-end row all settle; request-count dimensions are unchanged and the token dimension stays off by default. The design record, the English and Korean rate-limiting pages, and the configuration assistant were updated.
• Added rate_limiting.limits.*.tokens_per_minute as a public dimension on every admission-time rate-limit bucket (#2395), completing the reservation lifecycle from #2419. A text-generation request (/v1/chat/completions, /v1/completions, /v1/responses, /v1/responses/compact, /v1/messages, /anthropic/v1/messages) reserves its whole-request input estimate plus its output limit (max_tokens, max_completion_tokens, max_output_tokens, else 4096) against per_client, per_api_key, the per_model entry matching the client-facing model, and global, then reconciles that reservation against the usage the backend reports: unused tokens return to the budget and an overage is charged. Each bucket's burst is one minute of the rate, a request larger than the whole burst is refused immediately, and the first dimension that cannot cover the request answers 429 with limit_type: "token_budget" and error.details.token_dimension naming it; a Redis-backed check that cannot reach Redis follows redis.fallback_to_memory (503 with Retry-After: 5 when fail-closed). Only the per_api_key budget is reported, on new x-ratelimit-tokens-{limit,remaining,reset} headers. per_backend.tokens_per_minute is ignored and config validate warns about it. New metrics are rate_limit_token_budget_tokens_total{dimension,outcome} and rate_limit_token_budget_refusals_total{dimension}, and skipped dimensions appear as rate_limit_skipped_total{dimension="token_budget_<dimension>"}. Streaming replies hold their reservation until the body ends; streaming settlement is #2420. The configuration assistant, the IDE rule files, two config templates, the example config, and the English and Korean rate-limiting pages were updated.
• Added the /v1/fine_tuning/jobs passthrough (#2358), so OpenAI SDK clients can create, list, retrieve, and cancel fine-tuning jobs through the router and keep its auth, rate limiting, and usage attribution. Create selects a backend from the request's model, list selects one fine-tuning capable backend and returns only the caller's jobs, and retrieve and cancel reach only the backend and principal that created the job through a per-router binding. Only openai and generic types on http/https are candidates, and a request no candidate can serve answers 400 fine_tuning_not_supported rather than a router 404. Job CRUD does not run request-parameter policy, guardrails, response caching, or the per-model generation budget, and each call is metered as a non-completion request with zero tokens.
• Added the prompt logging operator guide, docs/en/guides/prompt-logging.md (#2333). It covers what leaves the router by default, enabling capture, consent and the fail-closed window, redaction, retention, the webhook, Langfuse, and S3 sinks with their at-least-once delivery rules, the routes that are and are not recorded, and the metrics to watch. It states that Hub push and Hub viewing are not available yet. The design record now matches what shipped.
• Added per-backend forward_cache_control (#2343): on /anthropic/v1/messages, a request converted to the OpenAI wire format now keeps the client's cache_control breakpoints when the selected backend opts in. A breakpoint on system text, message text, image, or document text parts is copied onto the OpenAI-wire part the conversion produces, and a system block array keeps its per-block breakpoints instead of being joined into one string. The receive side of a peer router rebuilds the Anthropic block-array system and the image source.cache_control from those parts. The default is true for type: continuum-router and false for every other type, because strict cloud endpoints and most local engines reject or ignore the field; native anthropic and bedrock backends are unaffected.
• Added the prompt_logging.export.sinks runtime (#2328). While prompt_logging.enabled is true, each configured sink tails the record files on its own owner-only cursor (.export-cursor-<id>.json under directory) and delivers bounded batches off the request path, at least once, retrying with backoff. Every record passes redaction.patterns again on read. Sinks: webhook (JSON array, credentials through headers_env), langfuse (OTLP JSON to {endpoint}/api/public/otel/v1/traces with Basic auth from public_key_env and secret_key_env, because Langfuse sunsets /api/public/ingestion on Cloud on 2026-11-16), and s3 (one JSONL object per batch under a key derived from the file position, behind the new prompt-log-s3 feature in official release binaries). A sink with no cursor file exports only what is written after it starts, with the start stored at once so a restart skips nothing; backfill: true sends every retained record first. A cursor that cannot be read is retried without moving the position, and an unparsable one follows backfill with a WARN: the tail when false, the oldest file when true. An http:// endpoint on a host that does not look private draws a clear-text warning, and an s3 sink must set access_key_id_env and secret_access_key_env together or neither. New metrics: prompt_log_export_total{sink,result} and prompt_log_export_backlog_bytes{sink}. The record file sequence now continues above existing files and cursors after a restart instead of reusing a free lower number.
• Added fallback.fallback_policy.content_policy_fallback (#2319). When true, an input guardrail block on a non-streaming POST /v1/chat/completions request advances the dispatched model's fallback_chains instead of ending the request, so a strict policy on the primary does not reject a request a laxer hop accepts. The hop re-runs the hop model's own input gate on the client document before dispatch, counts against max_fallback_attempts, runs with its own fallback_timeout_multiplier window, takes a fallback dial permit, honors the key's allowed_backends and re-applies allowed_models, and passes the hop model's output gate before its reply is served. A block from the hop's gate is served and ends the chain. A hop-served reply is then judged by fallback.quality_check (and usage.quality_judgment when configured), which may advance the chain once more. A transport failure advances to the next chain member unless it is terminal (a replay: disabled backend the request may have reached, or a partially generated internal stream), and the primary's block is served when none can answer. A client models list overlays the chain for the request. The primary and hop passes fold into one request-level input decision, a served hop reports X-Fallback-Reason: content_policy, and fallback_content_policy_hops_total{result} counts triggered, hopped, no_chain, blocked_again, and budget_spent. The field defaults to false, keeping the terminal-block contract byte-identical, and streaming requests, the classifier egress gate, the responses bridge, web-search loop requests, the other ingresses, and a request that opts out of cross-model hops (disable_fallbacks: true or a fallback: within_variant experiment) keep it.
• Added fallback.default_chain, a catch-all fallback list for any model that has no explicit chain (#2348). Resolution stays model_settings[].custom_fallback_chain, then fallback_chains[model], then default_chain: an explicit entry always wins, and an unset list preserves the previous behavior exactly, with no chain and no extra allocation. The list is flat and shared, so the dispatching model's own name is dropped from its resolved list (a model never hops to itself) and the executor never follows a hop's own chain (the list cannot cycle). Names follow the fallback_chains rule and duplicates are refused. A hop target the requesting key's allowed_models does not permit is skipped instead of dispatched, logged once per (origin, target) pair, and counted in fallback_default_chain_hop_skipped_total{original_model,fallback_model}; a request whose every target is filtered behaves as if the model had no chain, and explicit chains keep their existing behavior. Because every unlisted model now counts as chained, a quality_check with an empty models list judges it and the response cache route stays disabled for it. config validate warns when a default-chain target is not served by any enabled backend, suppressed while an enabled backend discovers models at runtime.
• Added the opt-in alerting configuration section (#2356). With alerting.enabled: true and a webhook.url, the router pushes backend health transitions (backend_unhealthy, backend_recovered), circuit breaker transitions (circuit_opened, circuit_closed), and Hub-synced key budget milestones (budget_exhausted, plus one budget_fraction warning at a configurable fraction) to one operator webhook as a metadata-only JSON payload (event, occurred_at, router_id, subject, detail, suppressed). Delivery is fail-open: producers only try_send into a bounded queue (max_queue), a dedicated task performs the POST, a full queue drops and counts the event, retries use bounded exponential backoff, and a per event and subject cooldown coalesces repeats while the next event carries the suppressed count. Failures are logged without the URL and counted in alert_deliveries_total{event,result} beside alert_queue_depth, alert_dropped_total, and alert_coalesced_total{event}. The section is off by default and validated on every load: https only, no embedded credentials, the strict SSRF blocklist unless allow_private_network, and the resolved addresses are pinned. The two budget events require the control-plane build and are inert with a startup notice otherwise. A new webhook on hot reload applies to events queued after the reload, and the serving path starts the notifier (idle while disabled).
• Added the context_window_exceeded fallback trigger (#2307). With fallback.fallback_policy.trigger_conditions.context_window_exceeded: true, an input that exceeds the primary model's declared context window hops before dispatch to the first configured chain target whose own window fits it, for buffered and streamed requests alike, and judges that target on input size even when the request carries no max_tokens. The trigger defaults to false, which keeps the existing 400 context_length_exceeded. When no target fits, the client receives that same refusal naming the primary's window and estimated input tokens without a backend dial, and a primary with no known limit fails open. A dispatched hop reports X-Fallback-Reason: context_window_exceeded. Hub config sync accepts the key, and the hub must gate it on a router version that does. The router does not report the key in its config snapshot, because a hub whose observed-path registry lacks it rejects the whole heartbeat (#2844). Reporting can resume only once every supported hub, including the compat floor, has the key.
• Added the LeastCost selection strategy (#2335), which routes a request to the eligible backend with the lowest estimated cost among candidates serving the same model. Cost is the request's tokenizer-free input estimate plus its completion ceiling (max_tokens or max_completion_tokens on chat, max_output_tokens on Responses, max_tokens on Anthropic Messages) priced through the folded PricingInfo in USD per 1M tokens: the backend's model_configs[].metadata.pricing first, then the model-metadata catalog, with the long-context tier applied from the estimate. A candidate without pricing ranks last rather than as free, equal costs break on the output rate and then rotate, and a fleet with no priced candidate behaves exactly like RoundRobin. Dispatch paths without a parsed body (audio, tokenize, realtime, ACP) use the documented rate-only basis of one million input tokens and no completion. config validate warns when a model served by two or more backends has fewer than two priced replicas, and the configuration-assistant knowledge, CLI help, admin schema, sample config, and load-balancing guides document the strategy.
• Added the webhook guardrail provider (#2267). It POSTs {stage, model, texts: [...]} to an operator-run HTTP service and reads {decision: allow|block|transform, category?, confidence?, reason?, spans?}, so an in-house rules service, a kept-alive LiteLLM CustomGuardrail, or a customer-list, checksum, or company-dictionary check runs through one documented contract instead of impersonating a guard model, a chat endpoint, or a Presidio analyzer. transform masks the returned spans, whose offsets are Unicode code points; masking reuses the PII provider's overlap resolution and redaction path. phase: sanitize, on_error, timeout_ms, and a per-provider circuit breaker behave as for the other providers, the call uses the shared HTTP client with a bounded retry and a 1 MiB response cap, and endpoint is required while backend is refused.
• Wired opt-in prompt logging into the request path (#2320). With prompt_logging.enabled: true, each POST /v1/chat/completions (the A2A bridge included), POST /v1/responses, or POST /anthropic/v1/messages request, buffered or streamed, that passes the per-key consent gate appends one redacted JSONL record to prompt-log-<sequence>.jsonl under the new prompt_logging.directory (default ./prompt-logs, created 0700, files 0600, rotated and pruned by retention). The record carries the prompt, the completion and tool calls, the usage, the consent basis, and the outcome (ok, refused, backend_error, client_cancelled with the partial reply, or stream_errored); a fallback across hops writes one record for the hop that answered, and each content field is cut at 256 KiB with the cut fields named in a new additive truncated field. The gate runs after the policy-enforcement middleware and before any body is copied, so a disabled section, a denied key, or a cold Hub policy store copies nothing. Redaction runs once on the request task, with the built-in credential patterns and redaction.patterns, each match replaced literally with redaction.masked_replacement (a $ can no longer expand, and a zero-width pattern such as \b is skipped rather than corrupting the record). Writing is fail-open: a dedicated thread owns the file, the request path hands it the record through a bounded queue, and a full queue or a failed write drops the record and counts prompt_log_records_total{result} (written, queue_full, sink_error) without touching the request. /v1/completions, embeddings, and ACP are not captured yet. A response-cache hit writes no record. The section stays restart-class, and enabled: true no longer logs an inert-setting warning. A stream that ends early (client_cancelled, stream_errored, or refused) has its output trimmed back to the last whitespace, at most 4 KiB, before redaction, so a partial token cannot evade a pattern, and the cut field is listed in truncated, as is a field the 1 MiB masking bound cut. model is capped and masked like other client text, built-in patterns now match a credential that directly follows Hangul or CJK text, key_id names a Hub key in permissive mode, and a disaggregated request is recorded buffered and streamed, with an orchestrator refusal as backend_error. retention.max_age_days holds on a quiet router: the writer thread applies it hourly, the active file rolls at one eighth of the bound, and a file is removed early enough that no record outlives it, so a record lasts between about seven eighths of the bound and the bound.
• Added POST /admin/routing/explain, a read-only explanation of the routing decision for a model and an optional API key (#2467). For each candidate backend it reports whether a chat request would be routed to it or the first live-path check that excludes it: an inactive key, the key's allowed_models or allowed_backends, a disabled or internal: true backend, an open circuit, an unavailable Hub credential, an unhealthy backend, or an exhausted per-backend rate limit (with retry_after_ms). It is a dry run that consumes no rate-limit token, moves no circuit, and changes no health or usage counter, and it does not run the selection strategy, whose pick advances shared counters. An unknown model answers 404 model_not_found, an unknown key 404 api_key_not_found, and the response never carries a backend URL or key value.
• Served /metrics on a dedicated listener at metrics.port (#2388). The field was parsed with a default of 9090 but nothing bound it, while the shipped Kubernetes manifests and Helm chart scrape port 9090 and their NetworkPolicy admits the monitoring namespace only there. With metrics enabled, the router now binds a second TCP listener on metrics.bind_address (new; an IP address or localhost, defaulting to the host of the first TCP server.bind_address entry) and metrics.port that serves only the metrics path, behind the same metrics.auth; API, admin, and health routes answer 404 there. The listener starts with the server and drains on shutdown. A port that cannot be bound fails startup with an error naming metrics.port when metrics.port or metrics.bind_address is set, so a host where 9090 is already taken needs metrics.port moved, or metrics.port: 0, which turns the listener off. When neither is set, the router logs a warning and starts without the listener (#2845). Upgrade note: the router now opens port 9090 by default, including with no metrics: section, so a router on a host where that port is taken logs the warning and keeps serving /metrics on the main listener; metrics.port: 0 turns the listener off. A file saved from config show or the Admin API export of an earlier release carries port: 9090, which counts as set. A metrics.port equal to a main listener port starts no second listener, a Unix-socket-only bind without metrics.bind_address skips it with a warning (also from config validate), and an AppProxy ROUTER traffic_port equal to metrics.port is refused. /metrics stays on the main listener as well. Changing metrics.enabled, metrics.port, or metrics.bind_address on hot reload logs that a restart is required. The three Helm values variants that omitted metrics.auth (and therefore answered 401 to the unauthenticated ServiceMonitor) now set method: none like the default values file.
• Added q and limit filters to POST /v1/models/query and accepted capability as a spelling of capabilities (#2758). q is a case-insensitive substring match on the model id and display name, and limit (1 to 1000) caps the filtered result. Glob backends[].models entries now also count in the Hub alias local-collision check and in alias dispatch for a backend with no catalog entries, and matching them no longer allocates.
• Added backends[].supports_response_format, an operator declaration of whether a backend honors response_format (#2759). Absent keeps inference from the backend type. false routes a request carrying the field around the backend, refuses it with 400 when no candidate remains, and removes the field from supported_parameters on GET /v1/models. true has no effect on anthropic and bedrock, whose conversion drops the field, and config validate warns. continuum_response_format_inferred_dispatch_total{backend} counts candidates retained on inference.
• Added per-key prompt-logging consent on the router side (#2324). KeyEntry gains the optional prompt_logging_consent (allowed or denied), and the router advertises prompt_logging_consent_v1, keeps the value in the persisted key table, and exposes it on the request in both auth modes. A new gate decides capture: prompt_logging.enabled: false never captures, denied is never captured, allowed is captured even under default_consent: denied, and a key that states nothing or a request with no key follows default_consent. While a Hub policy store holds no policy (before the first sync, or after a restored key table expires), a key-authenticated request is not captured whatever default_consent says. No Hub sends the field until continuum-hub#1975. Prompt-log records gain an optional consent field (key_allowed or deployment_default), and PROMPT_LOG_SCHEMA_VERSION is now 2.
• Added the prompt_logging configuration section (#2301). It defines the opt-in capture switch, default_consent, capture scopes, redaction patterns, finite retention bounds, and an export placeholder. It is off by default and validated under every feature set, and export.hub_push and export.sinks load with a warning and send nothing. Sink endpoint values and redaction.patterns are masked in Admin config, config show --resolved, config diff, and Debug output, pattern errors never quote the pattern, a $ in redaction.masked_replacement is refused, and an unresolved ${VAR} in a sink endpoint follows the usual config validate warning.
• Added the request_presets configuration section and the preset request field on POST /v1/chat/completions (#2436). A preset bundles an optional model default with the seven canonical request parameters, and a request selects it by slug. An explicit request field wins, the preset wins over request_params.defaults, and request_params.overrides/limits still apply. The preset model is used only when the request carries no usable model. Presets are validated at load and hot reload: a slug is 1-64 characters of [A-Za-z0-9._-], at most 256 presets may be declared, a non-empty model (at most 256 bytes, no control characters) or params is required, and parameter bounds match request_params; unknown keys are rejected. An unknown slug answers 400 unknown_request_preset (naming a well-formed slug), a non-string value answers 400 invalid_type, null is ignored, and the field is never forwarded to a backend. The section is separate from request_params, so Hub config sync never overwrites it.
• Accepted glob patterns in backends[].models (#2357). An entry with * (prefix, suffix, or infix, for example gpt-* or openai/*) routes every matching model id to the backend, so a provider's new model is routable without a configuration edit. /v1/models never lists the pattern string: it lists the concrete ids the backend's own model list advertises that match it, and a backend without a live list (Bedrock) lists nothing for it. An exact entry on any backend (or a dated snapshot of one) wins over a pattern, overlapping patterns on two backends draw a load warning and a config validate warning, and an entry containing ** or a blank entry is a load error. Per-key allowed_models still matches the requested id, so a pattern never widens a key, and Continuum Hub backend sync exports a backend using a pattern as preview-only because Hub profiles carry model ids.
• Added the LeastBusy selection strategy (#2334). It selects the eligible backend with the fewest requests the router itself has in flight against it, read from the pool's per-backend in-flight tracker, so it works over any backend type and requires no engine statistics (where EngineLoad defers to routing.engine_load.base_strategy for a backend that publishes none). Candidates the tracker has never counted read as zero, and ties rotate through the pool counter so equally loaded candidates share traffic evenly. Accepts the least-busy, least_busy, and leastbusy spellings in selection_strategy, --selection-strategy, and CONTINUUM_SELECTION_STRATEGY.
• Added opt-in hedged requests (#2317). With proxy.hedge_delay_ms set (0 to 60000, unset by default), a buffered /v1/chat/completions or /v1/completions dispatch that has not answered within the delay is sent to a second admissible backend serving the same model; the first response wins and the other attempt is cancelled and recorded as a usage row with outcome: hedge_cancelled. Client streams, fallback-chain hops, embeddings, rerank, image generation, and replay: disabled backends are never hedged.
• Honored a per-request model fallback list on /v1/chat/completions (#2403). A models array beside model, OpenRouter's fallback form, is extracted at the ingress and the key is consumed, so no backend sees it; any error on the primary fails over to the next model in the list through the same machinery a configured chain uses (per-attempt retries, X-Fallback-* headers, per-hop timeouts, the fallback dial bound, the mid-stream relay), the response model names the model that answered, and usage is recorded under it. When model is absent, the first entry names the primary. Each entry follows the configured-chain name rule, entries are deduplicated in the client's order with the primary never re-attempted, and a list that does not parse is a 400 naming the offending entry. The list applies only when the operator has the fallback feature on: without a fallback section, or with enabled: false, it is consumed and ignored, and it is also ignored when the dispatched model is not the model the client named as primary, because a Hub decision or a smart-routing rewrite owns that choice. It overlays only the dispatched model's chain for the request; every other configured chain, the fallback policy, and mid_stream_enabled stay as configured, and the model's notify_on_fallback choice is kept. Every entry must be in the key's model allow-list (an entry outside it is refused before dispatch, not skipped), and disable_fallbacks: true takes precedence: the list is consumed and the request stays on its primary.
• OpenAI-style batch creation now accepts an input_file_id from the router's local /v1/files store (#2733). The router checks ownership, purpose=batch, the 200 MiB scan bound, and model restrictions before sending customer content, then streams the file once to the admitted backend and substitutes its provider file ID. Provider-owned IDs still pass through unchanged; native Anthropic inline batches reject local file IDs, and provider copies keep the provider's retention lifecycle.
• Added continuum-router usage, a token usage report for a running router (#2525). It prints one row per model (default), backend, API key, or user (--by models|backends|api-keys|users) from the GET /admin/stats* endpoints, ordered by tokens with a share bar, failures, tokens per second, and average latency. Backend rows carry no token counters, so they are ordered and barred by requests. --series appends the global daily token series (GET /admin/stats/series) as a chart split into prompt and completion tokens. --window applies to the models and backends dimensions only, since the per-key and per-user counters are all-time. Authentication, --json, --timeout, and the control-character handling follow continuum-router status.
• Added the logging.access section for the request logging middleware (#2402). log_headers turns the TRACE-level header line on or off (credential headers stay redacted), and exclude_paths replaces the default /health, /health/ready, and /healthz exclusions; each entry must start with / and also covers the paths below it. An absent section keeps today's behaviour. Body capture is not configurable here, and unknown keys in the section are refused at load.
• Added age-based retention for rotated process logs, logging.file.rotation.max_age_days (#2369). When non-zero, the file sink also removes matching log files whose modification time is older than that many days, at startup and after each rotation, on top of the max_files count cap. The open file and files that do not match the configured pattern are never removed, a modification time in the future is not treated as stale, and 0 (the default) keeps count-only retention. The value is validated at load (0 to 36,500).
• Exported backend_health_status{backend,url} as a numeric enum gauge (0=unknown, 1=healthy, 2=warming_up, 3=unhealthy) beside the legacy binary backend_healthy (#2315). The health state machine already tracks WarmingUp internally with accelerated 1s checks and a max-warmup deadline, but the metric collapsed it to 0, identical to a dead backend, so autoscaling and dashboards could not tell a backend loading weights from one needing intervention. The legacy gauge derives from the same status, so existing scrape configs keep working unchanged.
• Published capability metadata on GET /v1/models and POST /v1/models/refresh (#2417). Each entry carries context_length, max_output_tokens, pricing, and capabilities from the resolved model metadata when they are known, and omits each one otherwise rather than sending null. Without operator limits, context_length is the minimum of the windows the serving engines report. Chat models also carry supported_parameters, the sampling and logprob parameters every serving backend's conversion carries to the wire (temperature, top_p, max_tokens, presence_penalty, frequency_penalty, top_k, min_p, logprobs, top_logprobs), derived from the same per-backend table the chat ingress enforces rather than maintained per model. Clients that filter on these no longer need /v1/models/extended.
• Exported per-provider failure details for routing debugging (#2435). Every backend hop that answered with an error HTTP status records its status code and a redacted error summary into a rolling per-provider tracker, surfaced at the new GET /admin/provider-failures endpoint and counted as provider_failures_total{backend, status}. At most 20 failures per backend and 256 backends are retained, a backend removed from the pool leaves the view, and the summary is the sanitized copy the client error already carries (issue #2253, cut at 200 characters), so no credential or internal endpoint is stored. The metric's status label is the exact code from 400 to 599 and other outside it.
• Added the OpenAI-shaped POST /v1/moderations endpoint (#2336), so client.moderations.create() works against the router. Each input (a string, or an array of up to 32 strings or text parts) gets its own results[] entry with flagged, per-category booleans, and category_scores, classified by the configured guardrails: the deny and allow match lists, then every input-stage provider of the route the optional model selects, at the same thresholds as live traffic but regardless of the global guardrails.mode. Categories are the router's guardrail categories; provider names and reasons are never returned, and a provider's fail-policy verdict never flags text. The route needs the write scope. With guardrails disabled it answers 400 moderation_unavailable, a request whose every provider failed answers 503 moderation_unavailable, and an empty, oversized, or non-text input answers 400 (invalid_input or too_many_inputs).
• Added the per-request disable_fallbacks body parameter on /v1/chat/completions and /v1/completions (#2347). true skips every cross-model fallback hop for that request, before the first byte and mid-stream, buffered and streamed, and returns the primary model's error; retries and failover between backends serving the same model still apply, and other requests keep their chains. The field is removed before dispatch whatever its value, so a backend never receives it.
• Added the mock_response request parameter on /v1/chat/completions for testing and debugging (#2346). A non-streaming request that carries it is answered from the value without contacting a backend or writing a usage record: a string becomes the assistant content of one chat.completion choice with finish_reason: "stop", and a JSON object is returned verbatim. Authentication, request validation, and the key's allowed_models still apply, and any other type answers 400 invalid_type. A streaming request is dispatched normally, and the field is never forwarded to a backend.
• Applied Continuum Hub-declared per-model backend aliases (#2723). A Hub backend profile's model_aliases (up to 50 caller-facing names per allowed model) materialize as model_configs aliases on the Hub-owned backend, so a request for an alias is dispatched as the target model (an internal profile's aliases are validated but not applied) through the existing alias rule: selection runs over every user-routable backend serving the target, the upstream receives the target id, and /v1/models lists the alias. The router advertises backend_model_aliases_v1 in heartbeat capabilities whenever backend sync runs (backends.mode is not disabled). A snapshot is rejected as model_policy_invalid when an alias names a model outside the profile's allowed_models, exceeds 50 per model, is malformed, or collides with another Hub profile's alias or model or with a preserved local backend's model or alias. A restored snapshot whose alias now collides with a local edit still activates with a warning. Realtime audio capability lookup no longer stops at a backend model_configs entry without metadata, so an alias-only entry cannot hide model-metadata.yaml capabilities.
• Reported per-key rejection counts for Continuum Hub-issued keys in the heartbeat supply block (#2515). Each rejection of a key the router resolved against the Hub key table is counted per key and stage in rejections_by_key, cumulative since process start; the first 32 such keys keep their rows until restart, and rejections of any other Hub key are summed in rejections_by_key_other, which is always sent (0 included). Local keys, AppProxy keys, and unattributed rejections still reach the Hub only through rejected_requests, which is unchanged. The vendored continuum-protocol crate is re-vendored at lablup/continuum-hub 37d98a1aca96cbc5b708aece86f3bee1f74f7e3f, which also brings backend-profile model aliases (applied since #2723).
• Added the opt-in usage.ledger_export.max_files retention knob (#2428). When set, the oldest rotated export files (<path>.<sent_at_ms> and its -<n> variants) beyond the cap are removed after each rotation; the live file is not counted, failures are logged at most once a minute, and the push is never affected. Unset (the default) keeps every rotated file, so tamper-evidence consumers that want the full history are unchanged.
• Added a fail-closed option and an outage signal for Redis-backed rate limiting (#2424, #2408). rate_limiting.redis.fallback_to_memory defaults to true, the existing fail-open behaviour in which an unreachable Redis degrades every check to the replica's own bucket (a fleet of N replicas admits up to N times the configured rate). Set it to false to refuse instead: admission checks answer 503 (type: service_unavailable, Retry-After: 5) until Redis answers, and a per_backend charge refuses the dispatch like an exhausted backend. Both cases are counted on rate_limit_redis_unavailable_total{dimension, action} (action is fallback or refused), and a fail-closed refusal is not counted as a quota violation. Each Redis check is now bounded at 500 ms, so a partitioned Redis no longer holds requests for the OS connect timeout. Changing the field requires a restart.
• Added the per-key token budget (#2323). A key with max_tokens_budget (whole tokens, input + output) is refused with 429 rate_limit_error once its usage within the rolling budget_window (1d, 7d, or 30d, default 30d) meets the budget, in the Anthropic error envelope on the /anthropic mount. Admission reserves an estimate of each in-flight request's prompt until its response body ends, so concurrent requests cannot all pass a nearly spent budget. Only budgeted keys are tracked, with fixed memory per key, and a key that cannot get a window is refused rather than admitted untracked. With api_keys.persistence_file set, the windows persist to <persistence_file>.budget.json every 30 seconds and survive a restart; otherwise they are in memory. The windows are per replica. max_tokens_budget: 0 and any other window spelling are load errors, and a key without a budget is unchanged.
• Served the Anthropic Messages API at the server root (POST /v1/messages, POST /v1/messages/count_tokens) as root-level aliases of the /anthropic ingress endpoints (#2276). A client whose base URL is the server root (without the /anthropic prefix) reaches the same handlers with no configuration changes; GET /v1/models stays the OpenAI-compatible model list. The root paths carry the same API-key protection and read/write scope classification as their /anthropic counterparts, and the same path-keyed behavior: pre-route refusals (rate limits, Hub policy, concurrency ceiling, hop guard) use the Anthropic error envelope, responses carry the Anthropic request-id header, POST /v1/messages is charged against per_model rate limits and Hub cost-center budgets, and count_tokens stays exempt from both.
• Publish backend model_configs[].aliases on the model lists (#2248). Each alias an enabled backend declares is listed as its own entry on GET /v1/models, /v1/models/extended, /v1/models/{model}, and /anthropic/v1/models, with the canonical model's backends and metadata, exactly while alias dispatch would resolve it; it leaves the list on the refresh that drops the canonical model. A per-key allowed_models list must name the alias itself, and the backend allow-list, internal backends, and the routable-models rule apply through the canonical model's backends. A literally served name keeps its own entry, a model experiment's client-facing name stays with the experiment, and model-metadata.yaml aliases and the model_aliases slots are not listed. GET /admin/models reports declared aliases in the canonical entry's aliases.
• Reported the monetary cost in the usage block of a buffered /v1/chat/completions response (usage.cost plus usage.cost_details.upstream_inference_prompt_cost and upstream_inference_completions_cost, in USD) so a client can meter spend per request without joining usage records with pricing metadata offline (#2404). The cost is computed from the routed model's pricing metadata (the long-context tier when the prompt exceeds its threshold, and cached_input_discount on the cached prompt tokens), the same arithmetic the x-continuum-cost-microusd header uses. A model without pricing metadata, a response without a usage block, and a response whose backend already reported its own usage.cost are left unchanged. The stamped prices are the catalog's, visible to every client of the route.
• Completed the routing-decision explainer (#2467). POST /admin/routing/explain now previews the configured strategy over the eligible set and reports selected, deterministic, per-candidate rank, latency_ms, and score, plus virtual_model and excluded_models for smart routing. It accepts an optional request body for the prefix key and the cost estimate, and it previews the strategy without advancing the round-robin counter, consuming randomness, writing the hash-ring cache, or calling prepare() on a scorer, so two identical calls return byte-identical bodies. Eligibility is now the live path's own seam, and the response adds capability_mismatch, model_not_allowed_by_tier (from the applied Hub policy tier), credential_unavailable, not_in_backend_pool, unhealthy, and rate_limited with retry_after_ms. The new server.explain_response_header (default false, restart required) adds x-continuum-route: <backend> to ordinary API responses for debugging without admin access.
• Added GET /v1/quota, a read-only endpoint reporting the caller's own limits and remaining values (#2872). It reports the per-key request and minute-token buckets, the per-user and per-team request buckets, the in-flight count, the long-window max_tokens_budget, and Hub tier windows, each with its scope (fleet or replica). It charges no identity-bound bucket and creates no entry, so repeated polling and a key already at its limit both read correctly. The shared global and per_client margins are still charged at the /v1/models cost of 0.1 but are never reported, preserving the rule that shared levels stay off the wire (#1170). The route requires the read scope, carries no x-ratelimit-* or x-continuum-ratelimit-* header, and returns an empty document to an unrecognized caller, byte-identical to a recognized key with no configured limit, which extends the existing permissive-mode key-validity residual from per_api_key to max_tokens_budget.
• Added proxy.client_provider_preferences (#2396), off by default: POST /v1/chat/completions interprets the client's top-level provider object as OpenRouter-style per-request routing preferences. only, ignore, and max_price narrow the candidate set, order ranks it, and sort (price, throughput, latency) ranks it by live telemetry, all applied after backend visibility, the key's allowed_models and allowed_backends, and Hub and policy narrowing, so a client can never widen what the operator permits. A non-empty candidate set emptied by a filter answers 400 invalid_request_error naming provider.<field> instead of dispatching; an empty input set stays the usual 503. Unknown fields, and the two OpenRouter directives the router cannot honor (require_parameters: true, data_collection: "deny"), are refused rather than ignored, and the object is limited to 16 KiB, 16 names per list, and 1 to 128 bytes per name. With the switch on the router removes the key before dispatch, so an aggregator backend no longer receives it verbatim; an operator forwards a provider object deliberately with backends[].request_extensions.body.overrides.provider, which wins over the removed client key. x-backend still wins the first attempt. With the switch off the request path is unchanged.

Changed¶

• Hardened the token-budget path (#2899). A token-budget 429 now still carries the caller's own request-count x-ratelimit-* headers, which describe an independent bucket that admitted the request before the token check ran. A reloaded tokens_per_minute or tokens_per_day applies to the next request: the reload rebuilds the budgets and resets the bucket maps. A full token-bucket map fails open and records rate_limit_skipped_total{dimension="token_budget"} (minute map) or rate_limit_skipped_total{dimension="token_budget_day"} (day map) instead of skipping silently. A dimension with both periods configured reserves against them in one Redis round trip, and a denial by either consumes neither. RateLimitError is #[non_exhaustive] so a future refusal detail does not break downstream matches. The request-count per_api_key bucket is now keyed on a deployment-stable, domain-separated hash of the credential, so with storage: redis every replica charges one shared bucket; the existing request-count per_api_key buckets re-key once on upgrade and the old Redis keys age out at the configured TTL. per_model token budgets are admission-time dimensions, so unlike per_backend they are enforced and config validate does not warn about them.

• Recorded the four token-budget design decisions in the design record (#2898). The local tokens_per_minute/tokens_per_day dimensions keep one window per period, with no input/output split; the admission hold is exactly the whole-request input estimate plus the output ceiling, with no extra margin; a request that deduplication coalesced onto an in-flight leader is released in full while only the serving request settles its own reservation; and a response-cache hit charges nothing, its input tokens included. Each decision is pinned by a test, and no runtime behavior or configuration surface changed.

• Re-vendored continuum-protocol at lablup/continuum-hub d7903743f4942e33117bb23e4159b614d097a38e (#2817), so the crate is again a byte-identical copy with no LOCAL-AHEAD entries. A Hub at or after continuum-hub#1994 now delivers per-key prompt logging consent through the key table, and the operator guide says so. The router compiles against the new PII custom entity and PII provider row types but does not advertise guardrail_pii_provider_row_v1 or guardrail_provider_content_v2, so a Hub keeps projecting those statements to the legacy form. A statement that nonetheless carries PII custom_entities is declined as a whole and reported as guardrails_unavailable, because this router does not run them. make hub-compat gains a consent stage, scripts/hub-compat-consent.sh, that proves delivery against a live Hub (skip with HUB_COMPAT_SKIP_CONSENT=1).

• Emitted the refusal-log window summary on a timer (#2427). The WARN summary that reports how many refusals a 60-second window suppressed was written only by the next refusal after the window, so a burst that stopped mid-window never logged its final count. A background flush, started by the first refusal and run every 30 seconds, now writes the summary for every closed window that suppressed repeats; it shares the line format and the bounded table with the lazy path, and whichever runs first writes the summary, so it is never duplicated. An idle router writes no summary.
• Tracked the 256 most recently rejected key ids on request_rejected_total instead of the first 256 seen (#2514). When a rejection arrives for a new key id while 256 are tracked, the least recently rejected key is evicted: its counts are added to key_id="other" and its own series are removed, so totals summed over key_id stay exact and the series count stays bounded. Because the evicted key's lifetime count lands in other at once, increase() and rate() over key_id="other" jump on an eviction; alert on key_id!="other" series. A key that starts failing late in the process lifetime now gets its own series instead of sharing other until restart. The per-minute request_rejected_summary window still admits key ids first come, first served.

Deprecated¶

• Deprecated metrics.latency_buckets, which was parsed but never applied (#2514). http_request_duration_seconds always uses the compiled bucket layout (0.01 to 1200 seconds); continuum-router config validate and startup now warn at metrics.latency_buckets when the field differs from its default. Remove the field to silence the warning.

Fixed¶

• Restored the observed config report for fallback.fallback_policy.trigger_conditions.context_window_exceeded (#2956). Router v2.0.0 requires a Hub that registers the leaf in its observed-path registry (continuum-hub #2038), because an older Hub rejects the whole heartbeat over the unknown path. Without the report, a Hub that delivers the leaf digests it as desired while the Router's observation omits it, so the configuration can never converge and reads as permanent drift. The Router parses and applies the leaf as before; only the value-free observation is restored, and the English and Korean configuration pages were updated.
• Fit the Anthropic ingress's derived reasoning_effort even when no reasoning_efforts vocabulary is declared (#2321). The fitting step now resolves, in order, the backend's model_configs declaration, the shared model metadata layer, and the backend type's documented default vocabulary before forwarding the derived value unchanged. An openai or azure target uses the built-in OpenAI reasoning-family table, and the self-hosted OpenAI-wire types take low, medium, and high with xhigh sent as high. The chosen source and the fitted value are logged at debug level as vocabulary_source=backend_model_config|model_metadata|backend_type_default|passthrough together with backend, model, and effort_field. An operator declaration still wins, a target with no known vocabulary still receives the derived value unchanged, and a client-supplied reasoning_effort on /v1/chat/completions is still never rewritten. The English and Korean reasoning-effort architecture and API pages were updated.
• Closed four gaps in responses_tools.deny_resource_fields (#2930). The opt-in input_item_resource_ids entry refuses any input item carrying container_id, file_id, vector_store_ids, previous_response_id, or conversation at any depth, names the location as input[code_interpreter_call].container_id, and applies to POST /v1/responses/compact; it is off by default because a client replaying the model's own output resends those IDs, and a file_id the router resolves to a local file is accepted either way. The list also judges every nested tools list a forwarded tool declares, including a programmatic_tool_calling and a raw tool_choice such as allowed_tools. An allow or deny entry must now be printable ASCII, so an invisible look-alike such as a zero-width space is a load error instead of a name that never matches. A request the pass-through strategy refuses is no longer served from or stored to the response cache, so a reload that adds an entry takes effect against an answer cached under the old policy. The configuration assistant and the English and Korean configuration and API pages were updated.
• Fixed the Hub prompt-log push losing its backlog silently (#2880). A 404 now drops the backlog only when the body is the ingest handler's own {"error":"prompt-log ingest disabled"}; any other 404, such as a proxy misroute, retries so the records stay on disk. The records a closed channel jumps over are counted as dropped (an upper bound), single-record isolation requests are paced, and a wholly refused batch pauses the push for one interval. The dead clear-text warning was removed, because such a hub_url is refused when the configuration loads, and a persistent redirect is retried and named once per process with its status.
• Kept unknown fields on a replayed message for native backends (#2896). A pass-through backend now receives a message's id, status, and any other field the router does not model, including on a role-and-content item without a type, on streaming and non-streaming requests. A backend may reject a replayed id it does not know, for example under store: false, as it already may for function_call. Conversions ignore these fields and their output is unchanged. The fields count toward the input size limit, and the input guardrail reads them. The same reading now also applies to a message the parser keeps verbatim, whose unmodeled fields previously reached a native backend uninspected.
• Fixed two prompt-log consent gaps (#2879). Export now decides consent on the key id as captured, before redaction masks it, so a redaction.patterns entry or the built-in sk- pattern that matches a key id no longer turns that key's records into consent_key_absent; only the masked id is written or sent, and the Hub push refuses a record whose id masking changed, counting it as rejected without a request and logging one warn without the id, because the Hub matches the id against its own table. With no Hub policy store, a presented token that does not authenticate is no longer captured under default_consent: an unknown token, a deleted or rotated key's old value, and a disabled or expired key stating allowed or nothing all write no record, so a client that sends a placeholder token must send no token to follow default_consent. With a policy store attached, only an inactive local key's denied still carries over, so a valid Hub key stating allowed stays captured. The English and Korean prompt-logging guides, design page, architecture pages, and the consent design record were updated.
• Fixed Gemini native /v1/chat/completions requests with only built-in tools hanging or failing upstream (#2890). The native body now carries toolConfig only when the request declares functions, so tool_choice with only web_search_options or code_execution is validated but not enforced, instead of reaching Google as a function calling config that answered with a 400, an empty reply, or no reply at all.

• Fixed prompt-log export startup and delivery gaps (#2848): a live writer no longer hides an implausible cursor, past-EOF cursors retry from a complete-line boundary, disabled capture keeps retention active, S3 credential endpoints are validated and pinned per delivery with bounded DNS checks, and S3 service-side request faults such as OperationAborted keep retrying. The global URL validator now rejects 6to4 and Teredo addresses that embed blocked IPv4 endpoints.

• Documented rediss:// trust for private CAs and made a rejected Redis server certificate visible in the logs (#2865). A connection failure caused by certificate verification now adds a hint naming SSL_CERT_FILE to the record's error text in the response cache, rate limiter, Hub tier minute limits, KV cache index, and AppProxy subscriber, and the rate limiter logs that case as a warning instead of debug. Fallback behavior is unchanged and verification still cannot be disabled. SSL_CERT_FILE replaces the system trust store, on Linux also for HTTPS backends and the Hub connection, so the deployment guide adds a private CA through SSL_CERT_DIR alongside the system bundle, and the Helm chart gains extraVolumes and extraVolumeMounts to mount it. The KV listener container smoke workflow now asserts the image carries a non-empty CA bundle.

• Fixed rediss:// Redis URLs, which passed config validate and then failed at start-up because the client was built without TLS (#2747). Every Redis consumer (the response cache, Redis rate limiting, the KV index, Hub minute limits, and AppProxy events) now connects over TLS and verifies the server certificate. A deployment that followed the documented rediss:// form and was silently counting minute limits per router now shares them.
• Stopped exporting backlog for a key whose prompt-logging consent was withdrawn (#2821). A sink now re-checks each record's key between the read and the delivery: denied, or a key stating nothing under default_consent: denied, is skipped as prompt_log_export_total{result="consent_withdrawn"}, a key no table holds is skipped as consent_key_absent, and while a Hub table cannot state consent the sink holds at that record without moving its cursor. An enabled AppProxy ROUTER holds a record of one of its appproxy- keys the same way until the router has its first key snapshot, and a ROUTER key its table no longer holds is skipped as consent_key_absent. A local key source that failed to load holds every record with a key until a restart. Delivered records cannot be recalled, local files are not rewritten, and capture scope narrowing stays unenforced.
• Closed streaming relay terminal and split-frame correctness gaps across native Gemini, /v1/responses, /anthropic/v1/messages, responses_only chat bridges, and image generation (#2829). GeminiStreamTransformer now buffers partial SSE lines and split UTF-8 sequences across chunk reads via SseLineBuffer and Utf8Buffer, enforces MAX_SSE_FRAME_BYTES on each line, and flushes any unterminated final line through finish() at stream end. When frame_json_refused rejects a structure-heavy SSE frame on the /v1/responses chat, Anthropic, or Gemini source relays, the Anthropic OpenAI and Responses bridges, or the responses_only chat bridge and the stream subsequently ends at [DONE] or EOF without a valid finish or terminal event, the router now emits that surface's error terminal and records upstream_error with any usage observed before the refused frame instead of synthesizing response.completed or message_stop. On /anthropic/v1/messages, an oversized trailing SSE frame or upstream read error arriving after message_stop has already been emitted is now ignored with a warning so no trailing event: error is appended and the usage outcome remains completed. Image-generation SSE error terminals now wait up to 5 seconds (IMAGE_TERMINAL_SEND_TIMEOUT) for queue capacity instead of using try_send, so a full queue that drains one slot receives the terminal error without stalling indefinitely on a dead client.
• Kept a Hub key's denied prompt-logging consent through the capability upgrade window (#2818). A Hub strips prompt_logging_consent from the key table of a router whose stored capabilities omit prompt_logging_consent_v1, so an already-enrolled router upgraded to a build that advertises it applied a stripped table before its first heartbeat and captured a denied key under default_consent. The router now treats a key table as carrying consent only when it was requested after a heartbeat the Hub accepted; until then key-authenticated requests are not captured, as in the no-policy window. The first heartbeat is sent at startup instead of after one interval, the policy is requested again as soon as the first heartbeat is acknowledged, a stream session opened before that reconnects, and a consent_confirmed flag in the persisted key table keeps a restored table's standing (a file from an older build restores as carrying no consent). A Hub that accepts heartbeats but sends no consent opens the gate and its keys follow default_consent, with one info line; a router whose heartbeats never succeed logs one warn and keeps key-authenticated capture closed. Under unmatched_key: fail_open, a presented token that no table recognizes is no longer captured.
• An export cursor that cannot be read at startup no longer stops the router (#2820). After 5 retries, 200 ms apart, the process serves traffic with no capture and no export, logs one ERROR, and sets prompt_log_capture_disabled{reason="cursor_unreadable"} to 1; restart after fixing the file. A directory that cannot be read or created still stops startup.
• Attributed rejected SSE event-name warnings and their 60-second suppression windows per serving backend (backend = %backend, or unknown when no relay backend is in scope) instead of a process-wide bucket, escaped and capped rejected image-generation SSE event names at warn with backend attribution, removed the unreachable raw - data: {} payload interpolation from the Anthropic streaming fallback warning, and stripped backend URLs (without_url()), bounded upstream error-body reads (read_bounded_upstream_error_body), and capped logged error bodies (cap_error_body_for_log) on the ACP prompt handlers (#2828).
• Corrected ACP session/prompt failure outcomes and bounded stalled session/update sends (#2827). A backend SSE stream that reaches EOF with neither a finish_reason nor [DONE] now fails with -32603 (Backend stream ended before the reply finished), leaves the session history unchanged, and records upstream_error with any usage seen so far instead of completed. A request total timeout that expires while reading a streamed or buffered response body now returns Backend stream timed out with {"timeout":"total",...} and records timeout_mid_stream once a body chunk was seen (timeout_first_byte otherwise) instead of upstream_error. Backend selection failures (-32602 unserved model and -32603 no available backend) now record one failed row under not_dispatched with no attempt counted, while a cancel before dispatch and an overlong model name still record nothing. Each session/update send now races session/cancel and timeouts.request.streaming.chunk_interval, so a client that stops draining its transport no longer holds the prompt open.
• Warned in ContinuumRouter::into_router() when control_plane.enabled is true and the control-plane usage or batch status receiver is still unclaimed, gated BatchManager and the batch status channel on the same control_plane_agent_runs flag as the usage channel, and honored ContinuumRouterBuilder::without_control_plane_agent() in serve() by skipping agent startup with a warning (#2826).
• Capped model names on the Anthropic Messages ingress, in StatsCollector, and in acp.default_model validation at 256 bytes (#2825). POST /anthropic/v1/messages, POST /anthropic/v1/messages/count_tokens, and their root aliases (POST /v1/messages, POST /v1/messages/count_tokens) now reject a model longer than 256 bytes before alias resolution or backend selection with 400 invalid_request_error (Model name exceeds maximum length of 256 characters) without echoing the value. StatsCollector now folds any model name longer than 256 bytes into unknown in per_model, per_api_key_model, and per_user_model while still counting the request, and drops overlong model keys when restoring a snapshot with a single warning naming the dropped count. acp.default_model values longer than 256 bytes after trimming are now rejected during configuration validation and continuum-router config validate.
• Bounded the per-stream SSE relay queue, the process-wide streaming buffer footprint, and accumulated A2A/ACP reply text (#2824). Each SseTx relay queue now has a 256 MiB ceiling (2 * MAX_SSE_FRAME_BYTES), and active SseParser buffers, SseLineBuffer buffers, and relay queues share a process-wide byte budget configured by server.max_stream_buffer_bytes (default 4 GiB, minimum 1; 0 is rejected at parse time, and changing it requires a restart). When a slow reader reaches the per-stream queue ceiling or exhausts the process budget, SseTx::send logs one warning, drops further client frames without pacing the upstream reader (always admitting one frame when the queue is empty), and returns Ok(()) so the relay keeps reading the upstream to its terminal usage event; DoneSentinel omits the trailing data: [DONE] frame on an overflowed stream so a truncated stream cannot appear complete. An SseParser buffer that exceeds server.max_stream_buffer_bytes ends the stream with SseLimitKind::ProcessBudget. Accumulated reply text in the A2A streaming task store and in ACP streaming and buffered session/prompt handlers is now capped at 64 MiB (MAX_ACCUMULATED_REPLY_BYTES), failing an oversized A2A task with Upstream reply too large and an oversized ACP prompt with JSON-RPC error -32603 (Backend reply exceeds the size limit) and upstream_error usage without mutating session history.
• Failed closed on structure-heavy SSE frames at streaming guardrail gates and bounded frame parsing in the Anthropic and Gemini stream transformers (#2823). An SSE frame with more than 1 MiB of structural JSON bytes outside strings cannot be parsed into a serde_json::Value without multi-megabyte or gigabyte heap expansion (#2795). Streaming output gates, stream usage tracking, and the Anthropic and Gemini stream transformers now route every frame JSON parse through parse_frame_json. When frame_json_refused trips at StreamingOutputGate::observe, the router logs a warning with the model and category = "guardrail_frame_uninspectable" without logging the frame: an active enforcing gate (mode: enforce with streaming_mode set to buffer_full or chunked) marks the stream blocked and returns StreamGateStep::Block so the relay emits its wire block terminal, drops any held text or deferred /v1/responses mirror events, stops reading, and records a guardrail_blocked usage row with the last usage reported before the refused frame (or the router's delta count); an inactive gate (mode: monitor, or mode: enforce with streaming_mode: passthrough) forwards the frame verbatim without building a JSON Value, and monitor mode's end-of-stream check runs without that frame's text. In the Anthropic and Gemini stream transformers, a structure-heavy frame is skipped or forwarded unparsed instead of calling serde_json::from_str.
• Hardened prompt log export after the #2328 security review (#2810). The strict SSRF validator behind Admin API backend URLs, Converse image fetches, client-supplied file URLs, and the export sinks now also refuses the shared address space 100.64.0.0/10 (carrier-grade NAT and the 100.100.100.200 metadata address), 198.18.0.0/15, 0.0.0.0/8, multicast, reserved, and NAT64-embedded addresses, so an admin-created backend on one of those ranges fails validation where it passed before; backends in the source configuration do not go through this validator and are unaffected. The export sinks resolve their endpoint before every delivery, refuse a name that does not resolve, and connect only to the addresses that passed, so a rebinding name cannot move a delivery; an S3 sink applies this only when an endpoint is in use, and also checks AWS_ENDPOINT_URL_STS and AWS_METADATA_ENDPOINT at startup and counts them in its cursor identity. Record masking, span building, and serialization run on the blocking pool; a batch delivered in part moves the cursor past the delivered prefix; retry delays carry jitter; a cursor file is read without following symlinks, without blocking on a FIFO, to 64 KiB at most, and a cursor beyond the record files or past its file's end is treated as corrupt or clamped; only regular files under their canonical name are exported; outcome and truncated are masked on read; bucket and prefix are redacted in Debug; a group- or world-writable capture directory warns at startup; an S3 bucket name with .. or an IP shape is refused; one half of the ambient AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY pair is named at startup and by config validate; the S3 credential clients keep their one second connect timeout; and every tracked streaming tool call is cut at a token boundary once a call past the cap stops collection.

• Bounded the SSE parser so a backend cannot exhaust router memory or burn quadratic CPU with one stream (#2795). A line or standard-mode event over 128 MiB, or an event: name over 1 KiB, now ends the stream with that relay's normal error terminal and an upstream_error usage record, and the parser scans in linear time. The image relay's 10 MiB limit becomes the same 128 MiB, and its queue gains a per-stream byte budget of 128 MiB. The A2A relay fails the task with Upstream stream frame too large, and a buffered dispatch stops reading on a breach. A frame with more than about a million structural JSON bytes outside strings is not parsed: a pass-through relay forwards it, a conversion relay skips it as an unparseable frame, and its usage falls back to the missing-usage path. Long strings, such as base64 images, keep their usage. These limits bound one frame, not a stream: there is no global byte budget, the relay queue has no byte bound for endless small frames, and the guardrail gates still parse every frame in full. A line now ends at the first of \r, \n, or \r\n, as the SSE specification defines, so a bare \r no longer merges two fields, and a blank line clears a pending event name

• Stopped a malformed upstream SSE event name from aborting the router process (#2791). A name holding \r or \n, taken from a raw event: line or a payload's type field, made axum panic, and the release profile's panic = "abort" turned that into a process exit that ended every in-flight request. The frame now goes out as event: unknown, and the router logs the capped, escaped name at warn once per 60-second window with a suppressed count, and at debug for repeats. Consumers that dispatch on the event name, including the router's own Responses conversions, drop such a frame. A \r-separated event: line can still lose its data: line in the parser, which is a separate defect.

• Aligned two model-name checks with routing (#2792). A Hub model alias that is the dated form of a preserved local models entry (or the reverse) is now rejected as a local collision, and an entry that matches every name (* or models/*) no longer holds a name back from alias dispatch, so one match-all backend cannot turn dispatch off for the whole router.

• Reject chat inputs whose typical token estimate reaches the routed model’s effective context window before backend dispatch (#2326), including streaming requests without an explicit output budget. Unknown or zero limits remain permissive. Configured per-backend limits override cached engine reports, and the smallest known serving window applies after routing and compression. Operators whose metadata understates the real backend window must correct it to restore requests previously accepted beyond that advertised limit.

• Published the same client-facing model names to the Hub inventory as to the model listings (#2434). The heartbeat's router-level models_served came from the raw aggregation, so backend model_configs[].aliases and model-experiment names, which GET /v1/models advertises and dispatch resolves, were absent from the Hub's endpoint model observation, and the portal marked a served model as not served. Per-backend attribution stays on the raw catalog.

• Treated CONTINUUM_DEV_MODE as a boolean (#2361). The built-in development API keys, one of which carries the admin scope, were loaded whenever the variable was set, so CONTINUUM_DEV_MODE=false or 0 turned them on in a release build. Only true or 1 (case-insensitive) now enable them; false, 0, an empty value, and any unrecognized value (which logs a warning) keep them off. An unset variable keeps the previous build-type default: on in debug builds, off in release builds.
• Bounded stalled ACP session/prompt streams and metered ACP prompts (#2762). A streamed prompt now fails with JSON-RPC error -32603 and data such as {"timeout":"chunk_interval","timeout_ms":1000} when the backend exceeds timeouts.request.streaming.first_byte (from dispatch to the first body chunk) or chunk_interval (every later gap), instead of holding the session until the total budget. Every prompt that reaches a backend now records one stats row and its token counters, with outcome completed, length, client_cancelled, timeout_first_byte, timeout_mid_stream, or upstream_error. Streamed prompts request stream_options.include_usage; without a usage block the record carries the router's input estimate with token_source: none.
• Stopped queuing undeliverable ACP usage in --mode stdio, and bounded ACP model names and per-model statistics (#2790). With control_plane.enabled: true, stdio mode runs no control-plane agent, so nothing drained its usage channel and every record logged a warning once the channel filled. It now opens no usage channel or usage write-ahead log, logs one warning at startup, and keeps usage in local statistics. session/prompt now rejects a model name longer than 256 bytes before backend selection, with JSON-RPC error -32602 and data {"field":"model","code":"model_too_long"}. On every ingress, per_model statistics now track at most 10,000 names: a new name past the bound counts under unknown, and names already tracked, including those in a restored snapshot, keep their own rows.
• Enforced the per-key allowed_models list on the realtime WebSocket upgrade (GET /v1/realtime, GET /realtime) (#2695). A key restricted to some models could open a realtime session on any other audio model the catalog serves, while the same model on /v1/chat/completions answered 403. The handshake now runs the shared enforce_model_access gate on the requested name before the reserved-variant check, alias resolution, and the upgrade, so a refused key gets 403 permission_error, never completes the handshake, and dials no backend; an alias of a model outside the list is refused as well.
• Fixed make docker-run so the container reaches the advertised backends and is reachable on host port 8000 (#2646). The target passed ROUTER_BACKENDS, which nothing reads, and published container port 8000, where nothing listens. It now passes CONTINUUM_BACKEND_URLS, the backend override the configuration loader reads, maps host port 8000 to the image's port 8080, and adds --add-host=host.docker.internal:host-gateway so host.docker.internal resolves on Linux Docker Engine as it does on Docker Desktop.
• Flushed unterminated final lines from image-generation and KV-index SSE streams, preserving split UTF-8 sequences in KV events (#2751).
• Applied one file-access policy to Responses, Anthropic, and chat-completion file references, so an admin-scope key can reference an admin-uploaded file only when files.auth.admin_can_access_all is enabled (#2751).
• Propagated one managed tracing context from native Anthropic Messages ingress to TCP, Unix-socket, and Bedrock dispatches without duplicate operator or client trace headers (#2751).
• Bounded every asynchronous wait in the Dynamo broker tunnel integration test so a stuck dependency reports its expired operation instead of hanging the target (#2751).
• Kept a response_format request off the backends that cannot carry it (#2438). Only the OpenAI-compatible wire and the Gemini conversion carry a client response_format through; the Anthropic Messages and Bedrock conversions drop it, so a structured-output request could be dispatched to such a backend and answered without the requested format. A chat completion that carries response_format is now routed only to candidates whose conversion keeps it (across the routed model and its fallback models), and is refused with 400 (param: response_format, code unsupported_request_parameter) naming the routed model when no candidate of that model keeps it. A request without response_format routes as before. GET /v1/models lists response_format in supported_parameters when every serving backend carries it.
• Metered a streamed /v1/completions request (#2331). The legacy ingress answers stream: true from the buffered funnel, which returned the backend's SSE text without a usage block and handed the stats seam a body it could not parse, so no usage was recorded. The router now asks the backend for stream_options.include_usage: true (other client stream_options fields kept) and meters the usage-only chunk it sends. The reply follows the client's own include_usage like a chat stream: no usage unless the client set it, and then only on the trailing usage-only chunk. The reply is still buffered and served as text/plain; a streaming relay for this ingress is a separate change.
• Shaped every /v1/chat/completions stream to the client's stream_options.include_usage (#2271). A client that did not set include_usage: true receives no usage at all: the chat ingress removes usage from every chunk (null included) and drops the trailing usage-only chunk, after the usage record, the response cache, and the guardrail gate have read the stream, so metering is unchanged. A client that set include_usage: true receives usage only on the trailing usage-only chunk, the shape the OpenAI stream sends: a chunk that carries usage: null passes as it was, and a chunk that carries a non-null usage with choices has the usage removed (the running usage a per-chunk engine reports). When such a stream reaches [DONE] without a usage-only chunk (an engine that reports usage only on its last content chunk), the router writes one carrying the last removed usage, so the client still receives it once. Clients that accumulate chunk.usage per chunk no longer double-count tokens, and the per-model proxy.honor_include_usage_models opt-in list is deprecated and ignored: listing a model changes nothing, and the field still parses.
• Recorded POST /v1/responses/compact in request statistics and the usage ledger (#2690). The backend served and billed each compact request, but the router wrote no row, so compact traffic was missing from /admin/stats*, per-key and per-user usage, and the control-plane usage ledger. A served request now records one row with the backend's token usage, and a backend error, unreadable or unparsable reply, or connection failure records one failed row with no tokens. A request refused before dispatch (validation, the per-key model or backend allow-list, or a backend type without compact support) still records nothing.
• Refused a chat completion whose output budget is below the wire minimum, before dispatch (#2366). A max_tokens (either OpenAI spelling) of 0 used to be forwarded, and the upstream that rejected it named its own internal field instead of the one the client sent (max_completion_tokens: Validation error: range ...), so an SDK or IDE plugin that maps errors by the param or message-prefixed field showed and rewrote a field the client never sent, and auto-retry logic keyed on the offending field misfired. The router now answers 400 with the OpenAI envelope naming the field the client sent (max_tokens must be at least 1., with param set to that field), before any backend dial; a unit budget and above is forwarded unchanged.
• Stopped reporting a /v1/responses reply the backend cut short as completed (#2715). A chat finish_reason of length or content_filter now yields status: "incomplete" with incomplete_details.reason set to max_output_tokens or content_filter, on both the buffered response and the stream, which ends with response.incomplete instead of response.completed. An Anthropic backend's max_tokens and refusal stop reasons map the same way. Partial output is kept.
• Stopped turning an empty Anthropic system (system: [], or blocks with no text) into an empty system message on conversion backends (#2721). The empty entry billed one extra input token on every request compared with omitting system. The OpenAI-compatible and Gemini conversions now add no system entry for it; the Bedrock, Anthropic, and Responses conversions already skipped empty system text.
• Stopped forwarding stop sequences on the Anthropic wire when the backend will reason (#2279). Backends match stop_sequences inside their hidden reasoning phase, so a stop string that appeared early in the reasoning text (common for short sequences such as "e") aborted generation before any user-visible content existed and the response carried content: null with finish_reason: "stop". The OpenAI→Anthropic transform now drops stop_sequences when the transformed request carries a thinking parameter or targets an adaptive-thinking model (Claude 4.6+) that reasons by default, and logs the drop at debug. A model that only thinks when asked keeps stop_sequences when no thinking parameter is present, and thinking: {"type": "disabled"} keeps them on any model, so visible-content stop matching is unchanged there. The Bedrock Converse transform applies the same rule when additionalModelRequestFields.thinking turns thinking on. The trade-off: on a reasoning request the client's stop sequence is no longer enforced and visible content runs past it, instead of the completion being cut off.
• Delivered the action-safety verdicts where Claude Code reads them on streamed Anthropic Messages replies (#2682). With safeguards.action_safety enabled, the verdicts went out as a standalone safeguard_results SSE event, which claude-cli skips, so every auto-mode session fell back to its own billed classifier and showed the "isn't eligible" notice. The verdicts now ride on the turn's last message_delta as delta.safeguard_results. A reply that proposes no tool use, streamed or not, now carries an available check with an empty tool_uses map, because a missing result also moved the session to the client's classifier. Without an evaluator nothing changes, and the /v1/responses Chat Completions bridge keeps its event before [DONE].
• Relayed a sanitized excerpt of a backend 4xx body the router does not recognize, instead of a bare per-status message (#2353). A backend that explained a rejection in an unfamiliar JSON shape reached Anthropic Messages clients only as Invalid request sent to backend; the body is now redacted and bounded like an extracted message and prefixed with Backend rejected the request: HTTP <status>:. The /v1/responses relays (Chat Completions conversion, Responses-only streaming, and pass-through) apply the same fallback. Empty, HTML, and binary bodies and every 5xx keep the generic message.
• Carried Gemini thought signatures through /anthropic/v1/messages (#2312). A signed Gemini function call now returns its signature on a thinking block placed before the tool_use (signature_delta when streaming), and a replayed block writes it back onto the function call, so a Gemini 3 function-calling loop no longer fails on its second turn. Other backends never receive the field.
• Enforced the per-key allowed_models list on POST /v1/responses/compact (#2686). The endpoint never checked it, so a key restricted to some models could compact a conversation on any other model the router serves, while /v1/responses refused the same request. The check now runs on the model name the client sent, before alias and experiment dispatch, guardrails, and backend selection, and refuses with the same 403 (permission_error, insufficient_permissions) as /v1/responses; a model outside the list answers that 403 whether or not any backend serves it, and an alias is admitted only when the key lists the alias itself.
• Fixed the Hub tier minute-limit store staying on a demoted primary after a failover (#2745). A managed failover can demote the old primary without closing its clients. Those pooled connections kept answering PING, so the recovery probe reported the store back while every script failed with READONLY, and each router counted on its own until it restarted. A READONLY reply or a dropped connection now discards the pooled connections, so the next call resolves the endpoint again, and the probe writes a short-lived <key_prefix>probe:<router id> key instead of pinging.
• Dialled the Hub tier minute-limit store ahead of requests (#2748). The pool dialled lazily, so the first call after start-up, or after a recovery, paid for the name lookup and connection inside the 100 ms store deadline and could send that router back to its own windows for a probe interval. The maintenance task now opens a few pooled connections at start and on every recovery, under its own 1 s bound, and probes through them.
• Stopped the ACP initialize handshake from creating a phantom session (#2430, #2457). The negotiated capabilities were held in a real store session that was never removed, so every initialize added a session to GET /admin/acp/sessions until the 1-hour TTL cleanup reclaimed it. The capabilities now live on the connection's handler context: session/new and the session/load capability gate read them from there, and session/set_mode or session/cancel sent between initialize and session/new answers "No active session".
• Kept the ACP session/prompt turn when a session/update notification fails to send (#2431, #2458). A transport failure on the notification used to fail the whole prompt and skip the history update, although the backend had already answered and billed the turn. The failure is now logged, later notifications for the turn are skipped, and the accumulated reply still lands in the session history.
• Derived the ACP prompt's backend timeout from the timeouts configuration instead of a fixed 300 seconds (#2425). A streamed prompt uses the streaming.total budget and a buffered one uses standard.total, read from the hot-reloadable timeout snapshot, so a tighter limit is enforced and a longer one no longer cuts long completions at five minutes.
• Made ACP session/cancel abort the in-flight backend request (#2418). The prompt handler used to await the whole backend response, so a cancel only stopped relaying a body the upstream had already generated and billed. The request and the incremental body read now race the session's cancellation token, and a cancelled prompt reports stopReason: "cancelled" without adding the stopped reply to the session history.
• Kept System One choice options in the order the client sent them (#2453). POST /v1/decisions re-sorted a choice question's criteria by option id before forwarding it, so a decision server that labels options by position saw a different question than the client wrote. The passthrough target and the emulated chat prompt and tool schema now receive the options in request order. The router's own built-in batteries (smart-routing domain, action safety) still send their options in id order, as calibrated.
• Fit the Anthropic ingress's derived reasoning_effort onto the vocabulary the serving model declares for the DeepSeek models (#2268). model-metadata.yaml and the 15-deepseek.yaml fragment now declare reasoning_efforts for deepseek-v4.1-flash (low, high, xhigh, max, the accepted set measured from the API's own rejection) and deepseek-v4-pro (low, high, max, its documented three levels). Claude Code's output_config.effort: medium now fits to high instead of being forwarded verbatim and rejected, which had failed every Claude Code request on DeepSeek models with a 400.
• Removed the unwired ServiceRegistry from src/services/mod.rs (#2300). Its health_check() always reported Healthy for four services it never registered or ran, and its initialize registered nothing; nothing outside its own unit tests constructed it. The Service trait and ServiceConfig stay, and HealthServiceImpl is unchanged.
• The published container image now starts without a mounted configuration (#2429). Its default command passes --config /etc/continuum-router/config.yaml, but no file was ever placed there, so docker run -p 8080:8080 ghcr.io/lablup/continuum-router exited at once. Dockerfile and Dockerfile.ci now write the binary's own --generate-config output to that path at build time and run config validate on it, so the build fails if the shipped default stops validating; a mounted file still overrides it.
• GET /v1/models/{model} now resolves the dateless form of a dated model id (#2577). The endpoint matched ids with raw string equality, so GET /v1/models/claude-opus-4-5 answered 404 while /v1/chat/completions routed the same name to claude-opus-4-5-20251130 through the date-suffix fallback. Both the cached and the aggregating lookup now try an exact match first and then the same date-suffix fallback the routing path uses; the response carries the matched catalog id.
• Resolved the Anthropic [1m] context-window marker that Claude Code appends to the configured model name (#2274). A request for <id>[1m] on /v1/chat/completions, /anthropic/v1/messages, or any other model-routed ingress now routes like <id>: the Continuum Hub tier allowlist and the per-key allowed_models list judge the plain id (an entry spelled either way matches), alias dispatch applies to it, and the upstream receives the plain id, while client-facing errors still quote the name the client sent. Previously the marked name failed the tier allowlist with a misleading 403 model_not_allowed. Only the exact [1m] suffix is stripped; model[1] and other bracketed names are unchanged.
• An error-terminated OpenAI-wire stream now ends with data: [DONE] (#2441): a chat-completions stream that ended on an error (an error-status handshake, a timeout, an upstream error terminal, the max-errors cut, an exhausted mid-stream fallback chain, or a response.failed / response.cancelled / error terminal on the Responses-bridged chat stream) stopped right after its event: error frame, leaving strict clients without the sentinel they close the stream on. Every OpenAI-wire relay body now ends with exactly one data: [DONE], appended only when the stream did not already relay one, and typed events render event: before data:. This replaces the earlier behavior of ending a cut Responses-bridged chat stream without a sentinel (#2122); the error event, not the sentinel, still carries the failure. Anthropic Messages and Responses API streams are unchanged.
• Stopped forwarding a null reasoning field on /v1/chat/completions responses from OpenAI-compatible backends (#2275). The router renames a non-null reasoning to the canonical reasoning_content but passed reasoning: null through, so the same model's responses alternated between the two spellings. A null reasoning is now removed from buffered message objects and streamed delta chunks, including next to an existing reasoning_content, and no reasoning_content: null is added. An existing reasoning_content is never overwritten.
• Made dropped tools visible when converting requests for Anthropic and Gemini backends (#2285). A declared tool whose type the conversion cannot carry (an OpenAI built-in such as file_search or web_search on Anthropic, or an unknown type) was removed from the upstream request with at most a debug line, and the native Gemini conversion did not log it at all. Each drop is now logged at warn with its type and counted on the new tool_conversion_dropped_total{target, tool_type} counter, whose tool_type label is limited to the OpenAI built-in names and other. Web search and code interpreter tools sent to Gemini are still mapped to its built-ins and are not counted. The request is still forwarded without the dropped tool.
• Made the library function core::tool_calling::transform_tool_message_to_gemini always produce an object functionResponse.response (#2314). Array content, numbers, and strings that parse to an array or scalar were passed through bare, which Gemini rejects with a 400; they are now wrapped as {"result": value}, and a string that parses to an object is still used as that object. The native Gemini chat transform already wrapped these values and now shares the same helper.
• Kept array-form tool results when converting chat history for Anthropic backends (#2305). A role: "tool" message whose content is an array of content parts, which the OpenAI schema allows, was sent upstream as an empty tool_result, so the model lost the tool output. The text parts are now joined with newlines; an array with no text part and an object or number value are sent as their JSON text instead of being dropped. String content is unchanged.
• Stopped the AppProxy ROUTER worker from advertising port 8080 when the router listens only on Unix sockets (#2433, #2459). With appproxy_router enabled, traffic_port unset, and only unix: entries in server.bind_address, the worker registered 8080, a port nothing listened on, so the coordinator's ingress sent traffic nowhere while the router looked healthy. That configuration now fails at startup with an error naming server.bind_address and appproxy_router.traffic_port. An explicit traffic_port is still honored, and the advertised api_port is now the port of the first TCP listener actually bound (including a unix:-first list with a TCP entry and an address passed to ContinuumRouter::serve) rather than one parsed from the first configured address.
• Sent the same tracing headers to Unix-socket backends as to TCP ones (#2528). The buffered chat, streaming chat, and Anthropic streaming chat Unix-socket transports always added the router's X-Trace-ID and forwarded the client's copy as well, so a backend could receive two conflicting values, and with tracing enabled they never injected X-Request-ID, X-Correlation-ID, or traceparent. They now drop the client's managed tracing headers when tracing is enabled and inject the router's set once; with tracing disabled the router adds none and a client X-Trace-ID passes through unchanged.
• Kept the event: type on a streaming image generation event retried after the client channel was full (#2530). The single 100 ms retry rebuilt the frame without its type, so a backpressured partial or complete event reached the client as an untyped data: frame. The retry now resends the exact frame the full channel handed back.
• Stopped corrupting multi-byte characters split across upstream network chunks on streaming relays (#2529). The Responses API relays (Anthropic, Gemini, chat-completions bridge, and pass-through), the Responses-only chat-completions bridge, and streaming image generation decoded each chunk on its own, so a Korean, CJK, or emoji character cut at a chunk boundary reached the client as two U+FFFD replacement characters. They now decode from raw bytes and hold a split sequence until the next chunk completes it. The shared SSE byte parser also no longer drops a whole chunk when an upstream byte is genuinely invalid UTF-8: that byte becomes U+FFFD and the events beside it, including [DONE], are still parsed.
• Attributed files uploaded through POST /admin/files to the synthetic owner __admin__ instead of storing them ownerless (#2644). With files.auth.enforce_ownership on, an ownerless file is served to every authenticated caller, so any key holding the files scope could read, download, or delete an admin upload once its id was known. Tenant keys now get 403 on those files; admin-scope keys still reach them through admin_can_access_all, and the Admin API is unchanged. The owner is reserved: no caller identity matches it, including a key whose user_id is __admin__, on the public Files API, its listing, and the Responses and Anthropic file resolvers. Files an admin uploaded before this change stay ownerless.
• Kept the uploaded filename on inline PDFs resolved from a file_id (#2411). The router used to discard the name the file was stored under and send {file_id}.pdf upstream, leaking the internal storage id as the document name. The inline file part now carries the stored filename with control characters removed, surrounding whitespace trimmed, and the length capped at 255 characters; {file_id}.pdf remains the fallback when no usable name was stored. The default and OpenAI file transformers behave the same.
• Rejected malformed file references on /v1/chat/completions before dispatch (#2409). An image_file or file content part whose file_id is missing or not a string used to drop out of file resolution silently, so the raw part reached the backend and the client got an upstream error about an unknown content type; a payload mixing one valid and one malformed reference was partially resolved. Such a request now answers 400 (file_resolution_failed) naming the offending field, for example messages[0].content[1].image_file.file_id. An inline file part that carries file.file_data and no file_id is not a reference and is forwarded as before.
• Ran the persistent metrics VACUUM on its own connection and off the snapshot task loop (#2303). Compaction used to hold the store's only connection lock and run inline in the task loop, so snapshot inserts and GET /admin/metrics/history queries waited for the whole file rewrite and snapshot ticks were skipped, leaving a gap in stored history. Snapshots now keep firing during compaction; an insert that meets the VACUUM write lock is retried on the next tick, and a compaction still running from an earlier slot is not started twice.
• Exported the rate_limit_* families on /metrics (#2578). Ten of them registered into the Prometheus default registry, which /metrics does not serve, so the limiter recorded checks, violations, skips, bypasses, and bucket counts that no scrape returned. Every family now registers into the scraped registry, beside rate_limit_redis_unavailable_total, which already did.
• Served /metrics when the config has no metrics: section (#2385). A missing section now means the documented defaults (enabled, path /metrics, fail-closed until metrics.auth is set) for the route, the request-recording middleware, the rate-limit exemption, and GET /admin/capabilities, instead of turning metrics off. The persistent metrics log still runs only with an explicit metrics: section, and ContinuumRouter::enable_metrics(false) still disables metrics when the section is absent.
• Served malformed JSON request bodies in the router's error envelope instead of the plain-text framework rejection (#2298). A body that failed to parse as JSON on an OpenAI-compatible route returned Axum's default plain-text 400, which JSON-parsing clients and SDKs could not read; it now returns the OpenAI error shape (invalid_request_error, code invalid_json) carrying the framework's rejection message, and the Anthropic shape on /anthropic/v1/messages, /anthropic/v1/messages/count_tokens, and their root /v1/messages aliases. A missing or wrong Content-Type (415) and a body that does not match the endpoint's request type (422) use the same envelope with their status unchanged.
• Enforced the per-key allowed_models list on /v1/images/generations, /v1/images/edits, and /v1/images/variations (#2506). A key whose non-empty list excludes the requested model now gets 403 insufficient_permissions from the router and the request never reaches a backend, as on chat completions and the audio routes; before, the image routes forwarded it. The gate judges the name the client sent and runs before alias and experiment dispatch, so a reserved experiment variant the key may not use answers the same 403 as an unknown name. The image generation log line for a per-backend rate-limit refusal now names a rate-limit store outage (fail-closed) apart from every backend being at its limit.
• Rejected a non-integer or negative max_tokens or max_completion_tokens on /v1/chat/completions and /v1/completions with 400 invalid_request_error naming the field (code: "invalid_type", or integer_below_min_value for a negative number) before dispatch, whether or not a request_params policy is configured (#2389). A string such as "64" used to be forwarded verbatim and could leave the generation uncapped until the client timed out. null and a missing field are still forwarded unchanged, so the backend's default applies; a request_params default or limit gives every request a router-side budget. /v1/responses and the Anthropic Messages ingress already refuse a non-integer budget when they parse the body.
• Rejected a /v1/chat/completions or legacy /v1/completions request whose model is missing, null, not a string, or blank with 400 invalid_request_error (param: "model", code: "missing_model", message you must provide a model parameter) instead of routing it to an arbitrary serving model with 200 (#2269). An AppProxy ingress-pinned request with no body model still routes, because the check runs after the ingress override supplies the model.
• Named the offending request field in error.param on the OpenAI surface when a backend validation error leaves param null but names the field in its message (#2363). vLLM writes max_tokens=1000000000 cannot be greater than ... and appends (parameter=<name>, value=...); either spelling now fills param when the name is a known request parameter (max_tokens, max_completion_tokens, temperature, and so on). An explicit upstream param always wins, 5xx responses are untouched, and the backend's message is still relayed as before.
• Answered an unmatched path on the Anthropic wire with 404 in the Anthropic error envelope ({"type":"error","error":{"type":"not_found_error","message":...}}) instead of the plain {"error": "..."} object, so an Anthropic SDK can type the error (#2432). This covers every path under /anthropic, the root /v1/messages/* subpaths, and the mis-prefixed Messages paths that get the base-URL hint, which stays in the message. Every other unmatched path keeps the plain object, and / and /v1/models still answer 200 for LM Studio.
• Kept the upstream status and sanitized message when a backend answers a round of the router-side web_search tool loop with an HTTP error on a non-streaming /anthropic/v1/messages request, instead of 502 web_search loop failed (#2339). The loop now answers through the same mapper as the single-shot path, so a context-window overflow returns 400 invalid_request_error with the backend's message, a native Anthropic error envelope is forwarded unchanged, and the upstream retry and correlation headers are relayed. Transport and loop-internal failures still return 502.
• Accepted the server_tool_use (web_search) and web_search_tool_result blocks of an earlier web search turn on Anthropic Messages requests to conversion backends, instead of rejecting the whole request with 400 Unsupported content block type(s) (#2340). Claude Code resends that history on the next turn, so every session that searched through the router's web search emulation against a self-hosted backend failed one turn later. The Chat Completions, Gemini, and Responses API transforms render the blocks as text in the same role and position (Web search: <query>, one - <title> (<url>): <snippet> line per result, or Web search failed: <error_code>). Malformed web search blocks and other server tools (web_fetch, code_execution) are still rejected, and native Anthropic backends still receive the blocks unchanged.
• Stopped turning backend reasoning into a thinking block on the Anthropic Messages ingress when the request did not enable thinking (#2311, #2400). A backend that reports reasoning_content on every completion (DeepSeek, newer vLLM) gave every response a leading thinking block, so clients that read content[0] as text broke. Thinking blocks and deltas now appear only for thinking.type enabled or adaptive, on the Chat Completions conversion path (buffered and streaming) and on the buffered Responses API path, whose streaming translator already applied the rule. Native Anthropic and Bedrock responses are unchanged.
• Rejected a malformed thinking: {"type": "enabled"} budget on the Anthropic Messages ingress (/anthropic/v1/messages and the root /v1/messages) with 400 invalid_request_error before any backend is dialed (#2342). A missing budget_tokens, a budget below 1024, and a budget at or above max_tokens are refused, the last one unless the anthropic-beta header carries an interleaved-thinking- token. Previously the router rewrote or dropped the field per backend and returned 200, and the native Anthropic transform raised max_tokens above the client's value. The check uses max_tokens after request_params is applied, so a policy that lowers it below the budget gets the same 400.
• Streamed session/prompt responses on the ACP stdio server incrementally instead of buffering the whole SSE body (#2414). The handler sent stream: true to the backend but read the entire response with response.text() before emitting anything, so time-to-first-token for an ACP client equaled the full completion time. The body is now read chunk by chunk, complete SSE lines (data: with or without a space) are parsed as they arrive, and each content delta is sent as a session/update notification immediately; only one incomplete line is held between reads, capped at 16 MiB. Every read races the session's cancellation token (#2418), a failed session/update send stays non-fatal for the turn (#2431), and a backend read error mid-stream fails the prompt with -32603 as the buffered read did.
• Routed the client's session/cancel JSON-RPC notification in the ACP stdio server to its registered handler instead of dropping it (#2379). The ACP spec sends session/cancel as a notification, and the stdio main loop dropped every notification, so a spec-compliant client's cancel never reached the session. A session/cancel notification now cancels the session its sessionId names (the current session when none is given) and installs a fresh token, so the session stays usable; the request form resolves sessionId the same way and answers -32001 for an unknown session. Only the methods the spec defines as client notifications run, and only after initialize; a request-only method sent without an id, such as session/new or session/prompt, is ignored. Notifications have no response, and handler failures are only logged. The stdio loop still handles messages one at a time, so a cancel sent while a prompt runs is processed after that prompt returns (#2380).
• Hardened eight residual rate-limiting, admin, metrics, and stats paths (#2718). A fail-closed Redis outage on the per_backend dimension now answers 503 service_unavailable with Retry-After: 5 instead of the capacity 429, and does not start a fallback chain. After a connection-level Redis failure the rate limiter skips Redis for 5 seconds and lets one request probe it, and its pool times out at 250 ms, so a partition no longer costs up to 500 ms per Redis-backed dimension on every request. The token bucket keeps fractional refill credit across frequent refills. Without hot reload, PUT/PATCH /admin/config/{section}, POST /admin/config/apply and /admin/config/import with a change to apply, POST /admin/config/rollback/{version}, and the backend mutation endpoints answer 503 instead of 200, and an unapplied rollback records no history version. /anthropic/v1/messages and /v1/messages share one endpoint label value. A streamed /v1/audio/speech response that fails midway is recorded as an upstream error, and one the client drops as cancelled.
• Re-split a hub-rejected (413) usage batch instead of dropping it (#2360). The hub answers 413 when a batch holds more records than its usage.max_batch_records; the router used to drop such a batch whole, losing up to usage_batch_max_records records of spend accounting and retiring their cost-center sequences. The records now return to the front of the queue and the next push builds half the batch, until the hub accepts or a single record remains (a single record the hub still rejects is dropped as before). Re-split records are sent before any spooled batch, so batches still reach the hub in router_usage_seq order. The abandoned batch is a chain_gap on the next one (the hub classifies it as gap, and its records arrive in the following batches under the same idempotency keys, so the local ledger export lists them twice). The halved size is kept for 5 minutes after the last 413 before the full size is tried again.
• Kept the in-memory token bucket's refill credit under contention (#2531). The refill window was claimed and the token count updated in two separate CAS steps whose failures were ignored, so a lost race could drop a window's credit, and a refused request never banked the credit it observed; per-client, per-key, per-user, and per-model limits could enforce below the configured requests_per_second under concurrent load. The window is now claimed exclusively and its credit is always written into the bucket, including on a refusal.
• Ran the in-memory rate limiter's idle-bucket eviction on the monotonic clock its refill already uses (#2422). The per-client, per-key, per-user, and per-model buckets recorded their last access with the wall clock, so a forward clock step (VM resume from pause, NTP correction) evicted live buckets and reset their quota, and a backward step kept idle buckets alive past their 10-minute idle TTL. Last access is now measured in monotonic microseconds since process start.
• Named the mounted Anthropic Messages paths in the 404 for a mis-prefixed Anthropic request (#2398). A POST to an unrouted path ending in /messages or /messages/count_tokens, such as /v1/v1/messages from an SDK whose base URL is <root>/v1, now answers 404 with a message that the API is served at /v1/messages and /anthropic/v1/messages (or the count_tokens twins) and that the client base URL should be the server root or <root>/anthropic. Other unmatched requests keep the plain message.
• Marked retryable error responses with x-should-retry: true and relayed upstream retry headers on the /anthropic conversion paths (#2341). Router-authored errors with status 408, 409, 429, or 5xx (529 included) now carry x-should-retry: true on /anthropic/v1/messages and on /v1/*; other statuses carry no hint, and the direct-mode shim refusal keeps its header set. When /anthropic/v1/messages converts a request for an OpenAI-wire or Responses-wire backend, and on web-search emulation and count_tokens, a backend error now relays the backend's retry-after, x-should-retry, request-id, and anthropic-ratelimit-unified-* headers as the native path already did, and an upstream x-should-retry replaces the router's value instead of adding a second one.
• Served the concurrency ceiling 503 and the router hop guard 400 and 508 on /anthropic/* routes in the Anthropic error format (overloaded_error, invalid_request_error, and api_error) instead of the OpenAI envelope, and gave refusals by the global and per-client rate limiters, the concurrency ceiling, and the hop guard the x-continuum-request-id, x-request-id, and (on Anthropic routes) request-id headers, so they can be quoted by id and match the event=request_rejected log line (#2511). OpenAI routes keep their bodies unchanged, and a response from inside the API routes keeps its single id.
• Recorded the client request ledger for legacy /v1/completions requests: the stream flag and the prompt text counts (a string, or each string in an array), which the chat, Responses, and Anthropic ingresses already note (#2444). A /v1/completions request was recorded with stream = false and no input counts, so a streamed legacy completion was metered as a buffered request without its input text.
• Extended the http_request_duration_seconds buckets past 10 seconds with 30, 60, 120, 300, 600, and 1200 seconds, so model calls longer than 10 seconds no longer land only in +Inf and p95 and p99 can be computed (#2260, PR #2469). The original nine buckets are unchanged, so existing dashboards and le selectors keep working.
• Counted every request the router rejects before backend selection per API key and stage, including rejections whose event=request_rejected log line the per-minute cap suppresses, and exported the count as request_rejected_total{stage, key_id} (#2256). At most 256 key ids are tracked individually; later keys share key_id="other", rejections without a resolved key use key_id="unattributed", and unusable key ids use key_id="invalid". The Hub usage ledger is unchanged because its protocol has no slot for rejected requests.
• List only models a caller can request (#2263, PR #2475). A model whose every backend the health checker marked unhealthy stayed on /v1/models, /v1/models/extended, /v1/models/refresh, and /anthropic/v1/models, so a client could pick it and the call failed. Such a model is now withheld unless an enabled fallback chain is configured for it (the outage retention of #1422 still keeps it reachable then); /v1/models/{model} keeps answering 200 with available: false. A catalog refresh that served a backend's list from configuration or a discovery fallback (the curated Anthropic list, the Codex configured list, the Bedrock and OAuth static lists) no longer marks that backend healthy; only a live answer does.
• Made the process-log fallback use the logging defaults: when the configuration file is missing or fails to load, the router now logs JSON lines at info, matching a configuration without a logging section, instead of single-line text (#2259). Both defaults come from one constant, and a configuration file that fails to load (or a missing --config file) now produces a WARN line in the process log saying the defaults are in use, before the router reports the load error on stderr and exits.
• Served rate-limit and Hub policy refusals on /anthropic/* routes in the Anthropic error format ({"type":"error","error":{"type":"rate_limit_error","message":...}}) instead of the OpenAI envelope, so Anthropic clients recognize a key's 429 as a rate limit (#2255). This covers the local rate limiter and the Hub tier, budget, cost-center, revocation, model-allowlist, and unmatched_key: reject refusals. OpenAI routes, status codes, and retry-after and rate-limit headers are unchanged.
• Reported which keys were rejected while the event=request_rejected log cap was suppressing lines (#2257). When a 60-second window that hit the cap ends, the router now writes one event=request_rejected_summary line with phase="window_closed" carrying the window's suppressed total, the top 20 keys by suppressed count in suppressed_by_key, the remainder in unlisted_keys and unlisted_suppressed, and suppressed_by_stage; the summary is written by the next request after the window ends. The notice at the first suppression now carries phase="cap_reached" and no longer reports a suppressed count that was always 1.
• Preserved the backend's own message on 4xx errors (#2253). /anthropic/v1/messages used to replace any non-Anthropic error body with a generic message such as Invalid request sent to backend, so a context-length overflow and an invalid parameter looked identical; it now returns the backend's message for OpenAI, Gemini, FastAPI detail, top-level message, and plain-text bodies, and a backend 400 maps to invalid_request_error instead of api_error (413 to request_too_large, 529 to overloaded_error). The Responses API streaming paths for Anthropic and Gemini backends and the Responses-only bridge keep the backend message on 4xx as well. Every relayed message, on these paths and on the OpenAI ingress that already relayed it, now has absolute URLs, inline credentials, bearer and provider API keys, key=value secrets, IP addresses, and host:port endpoints replaced by placeholders, with control characters removed and a 500-byte limit. 5xx responses keep their generic message, and the operator log still records the full backend body.
• Stopped relaying raw backend error bodies, bounded every upstream error-body read, and hardened the Unix-socket chunked decoder (#2512). The /v1/responses passthrough, /v1/responses/compact, the Responses-only bridge, Anthropic count_tokens, /v1/audio/transcriptions and /v1/audio/translations, OpenAI /v1/images/edits and /v1/images/variations, the Nano Banana image paths, and the native Gemini multimodal embedding path forwarded or echoed the backend's error body, in some cases even for 5xx; they now return the sanitized 4xx message (with the backend's type and code where present) or the generic per-status message, so a backend URL, internal host, or inline key no longer reaches the client. Upstream error bodies on every ingress, streaming, image, audio, Files API, and backend path are read up to 64 KiB instead of in full, and logged as an escaped field of at most 2000 bytes, so the operator log no longer records the full backend body as the #2253 entry above states. The Unix-socket streaming branches of /anthropic/v1/messages now read the backend's error body, so a 4xx carries the backend's message over a socket as it does over HTTP; every Unix-socket error-body read stops at Content-Length (unless the body is chunked) and gives up after 5 seconds, so a keep-alive engine cannot hold the request open. The chunked transfer-encoding decoder no longer allocates from the backend-declared chunk size: a size above 16 MiB or a size line over 1 KiB fails the stream, where FFFFFFFFFFFFFFFF used to abort the process and 40000000 allocated 1 GiB.
• Stored one row per series per value change in the persistent metrics log instead of one row per series per snapshot tick (#2290). Every snapshot used to re-insert the whole registry, so an idle router grew by series × (86_400 / snapshot_interval_seconds) rows per day regardless of traffic; a snapshot now persists only the samples whose value differs from the last stored one, and the first snapshot after a restart stores the baseline. An unchanged series is still stored once an hour, so retention pruning never removes the only row of an idle series. GET /admin/metrics/history returns these change points and hourly rows; a consumer carries the last value forward between them. A failed insert is retried on the next tick instead of being dropped.
• Enforced timeouts.request.streaming.chunk_interval on the thinking-pattern relay and the Unix-socket chat, thinking-pattern, and Anthropic relays (#2513). These relays bounded the gap between chunks with a fixed 60-second window, so a stalled stream on them waited 60 seconds even with a lower chunk_interval, and a model with a longer per-model chunk_interval was cut at 60 seconds. They now use the same per-model resolution as the plain chat relay, and a stall is reported as {"timeout":"chunk_interval","timeout_ms":<value>} instead of inactivity.
• Enforced timeouts.request.streaming.first_byte on the plain chat, thinking-pattern, and Unix-socket chat, thinking-pattern, and Anthropic relays, and both first_byte and chunk_interval on the relay that serves an OpenAI-compatible client from an HTTP Anthropic backend (#2656). The first five relays bounded the wait for the first chunk with chunk_interval and reported it as chunk_interval; the HTTP Anthropic relay had no read window, so a stalled upstream ran until the whole-request timeout (streaming.total) and ended with a generic stream error. All six now wait up to the per-model resolved first_byte for the first chunk and up to chunk_interval for every later gap. A silent upstream is reported as {"timeout":"first_byte","timeout_ms":<value>} and recorded with the timeout_first_byte usage outcome, as on the mid-stream fallback relay and the Bedrock and Gemini pipelines. With the defaults, the first-chunk wait on the plain, thinking-pattern, and Unix-socket relays grows from 30 seconds (chunk_interval) to 120 seconds (first_byte); lower first_byte to keep the old bound.
• Backpressured the usage WAL send path (#2367). When the line channel is full, a start or end line now waits up to 250 ms (WAL_SEND_TIMEOUT) for the writer to drain instead of being dropped at once, so a burst that outpaces one drain keeps its lines and the router_restart safety net keeps those attempts. A send that times out marks the log saturated: later lines are dropped at once (counted under usage_records_dropped_total{reason="wal_dropped"}) until a send finds room again, so a wedged state disk delays request handling by at most one wait. On the multi-threaded runtime the wait runs inside block_in_place, so other tasks keep running. A dead writer still drops immediately.
• Refused a chat completions or completions request that asks for logprobs (logprobs: true, the legacy integer logprobs, or a non-null top_logprobs) with 400 unsupported_request_parameter when a candidate backend conversion cannot carry them (#2364). The Anthropic Messages and Bedrock Converse conversions and the Chat-to-Responses bridge for responses_only models used to drop the fields silently, so a request routed (or fallen back) onto one answered 200 with choices that carry no logprobs at all. OpenAI-compatible backends keep forwarding the fields unchanged, and the rejection names the enabling logprobs field when both are present.
• Refused a chat completion whose output budget exceeds the model's declared limits, before dispatch (#2365). A max_tokens or max_completion_tokens above the limits.max_output or limits.context_window the metadata layer declares for the routed model (from model-metadata.yaml, a model-metadata.d/ drop-in, or the metadata download) used to be accepted and could hang with no bytes or error until the client timed out, while the backend kept generating or billing for an abandoned request. The router now answers 400 with the OpenAI envelope naming the field the client sent, the requested value, and the binding limit (for example max_tokens is too large: 200000. This model supports at most 128000 output tokens.). Limits resolve per serving backend: a backend's own model_configs entry takes precedence over the shared metadata layer, and the request is refused only when every candidate backend declares a limit it exceeds (the message names the largest such bound). A model with no declared limit on any candidate, and a zero window or output cap (an image model's "not applicable" marker), are forwarded and the provider stays the authority. Buffered and streamed requests are checked alike.

Removed¶

• Removed the duplicate BackendStatistics and LoadBalanceReport from src/services/backend_service.rs (#2337). Nothing imported them: every BackendStatistics user reads core::traits::BackendStatistics, and neither LoadBalanceReport nor the module was referenced outside its own tests.

Compatibility¶

• A build from main after #2956 reports fallback.fallback_policy.trigger_conditions.context_window_exceeded in its observed config. A Hub older than continuum-hub #2038 rejects the whole heartbeat over that unknown path, and a Hub that carries the registry withholds the leaf from a Router below v2.0.0, so such a Router's fallback section never matches the desired digest. The restore therefore ships only in v2.0.0, together with that Hub release; cutting a v1.36.x release from main after this lands either breaks the heartbeat against an older Hub or leaves the fallback section permanently drifted against a newer one.
• A build from main after #2877 that forwards file_search.vector_store_ids or a named code_interpreter.container now answers 400 unsupported_request_parameter until the operator sets responses_tools.deny_resource_fields: [] (#2891). This also breaks a client upgrading from a release: a released version dropped these fields and answered normally, and the same request now gets 400 on every backend type. The upgrade exposes no resource in exchange.
• A build from main after #2850 that forwards an unmodeled tool type such as mcp or image_generation now answers 400 unsupported_request_parameter for it until the type is added to responses_tools.allow (#2873). Released versions answered 422 for these types, so upgrading from a release opens nothing by default.
• Each http_request_duration_seconds series now exports six more _bucket samples (16 instead of 10, counting +Inf), so each label set stores 18 series instead of 12 including _sum and _count (#2260, PR #2469).
• A router started without any configuration file now writes its process log as JSON lines instead of single-line text (#2259). To keep text output, start the router with a configuration file that sets logging.format: pretty.

v1.36.1 - 2026-10-27¶

Added¶

• Added the opt-in request_extensions.sd_cpp_image_parameters backend setting for stable-diffusion.cpp sd-server (#2233, PR #2247). On /v1/images/generations, the router translates negative_prompt, seed, clip_skip, steps, sampler, scheduler, and cfg_scale into the <sd_cpp_extra_args> JSON block inside prompt, merges a valid existing block with request fields taking precedence, escapes < in generated JSON, and rebuilds the body from the original for every retry and fallback hop so the tagged prompt never reaches an ordinary backend.

Changed¶

• Breaking (security): Fail closed when admin.auth is absent, default a present auth block without method to shared-store api_key, and require explicit method: none for unauthenticated development access (#2234, PR #2242). admin.auth is now startup-owned: Admin section writes and full-document import, apply, or rollback refuse changes, while exports always omit it and ignore the legacy include_sensitive: true request with an audit record. Admin-authored new or changed backend, engine-metrics, and outbound OAuth destinations, including transient probes and full configuration write paths, reject private, loopback, link-local, non-public-at-validation-time, and Unix-socket targets; health overrides must remain same-origin and status lists are bounded. Trusted startup files may still declare local or Unix backends, and unrelated Admin edits preserve an unchanged private destination. The URL check resolves hostnames off Tokio workers and filters the addresses returned at validation time; deployments that need protection against a later DNS answer change must also enforce network-layer egress. Admin credential masking is Unicode-safe.
• Harden optional observability and Dynamo event surfaces (#2236, PR #2241). A configured /metrics endpoint now returns 401 when metrics.auth is omitted; deployments that deliberately allow anonymous Prometheus scrapes must set metrics.auth.method: none, while auth: {} selects bearer authentication and fails validation without bearer_token. Bearer and Basic credentials use constant-time comparisons. Dynamo direct-mode subscriptions accept dynamo.load.direct_allow_cidrs; when set, the router resolves each fleet-advertised publisher once and dials only an address in the configured worker networks. An unset list preserves existing in-cluster behavior, and config validate suggests adding it.

Fixed¶

• Dropped client temperature, top_p, and top_k on native /anthropic/v1/messages hops to Anthropic models or API versions that reject them, for buffered and streaming HTTP and Unix-socket requests (#2245, PR #2246). The client anthropic-version takes precedence over the backend version. Other providers keep the client values, Bedrock behavior is unchanged, and the original values are preserved for later fallback hops. Policy defaults and overrides for these fields still fail closed, and a fallback hop is refused when it would drop a policy-written value or forward a value hidden from a primary-hop limit.
• Masked every value in health_check.headers and tracing.otlp.headers in Admin reads, exports, history snapshots, and CLI and MCP redaction while keeping header names, restored masked values by header name on write, and redacted health-check header values from debug output (#2235, PR #2239).
• Masked Redis URL credentials in AppProxy startup logs and configuration debug output, and added a warning in both worker modes when the shared coordinator client uses plaintext HTTP outside loopback (#2237, PR #2238).
• Made make clean also clean the standalone crates under crates/ and the perf/ harness, which keep their own target/ directories.

Compatibility¶

• Deployments without admin.auth now receive 401 on /admin/*. Configure an authenticated method before upgrading, or set method: none explicitly for isolated development (#2234).
• Deployments that scrape /metrics without metrics.auth now receive 401. Set metrics.auth.method: none to keep anonymous scraping, or configure bearer or Basic credentials (#2236).
• Admin APIs no longer accept new or changed private, loopback, link-local, or Unix-socket backend destinations. Move intentional local backends into the startup configuration file (#2234).

v1.36.0 - 2026-10-27¶

Added¶

• Added per-backend anthropic_version for the 2026-10-21 API version and updated Haiku 5.5 compatibility and model metadata (PRs #2187, #2191, #2205).
• Added shared Hub tier minute limits across Router replicas (PR #2193).
• Added WebUI controls and Hub synchronization for per-key Anthropic user profiles (PRs #2203, #2221). Hub-synced profiles use the same attribution rules as local keys in both authentication modes.

Fixed¶

• Rejected provider batch submissions when their assigned cost center has exhausted its request, token, or sell-spend limit. Admission uses the completion-accounting assignment and does not charge usage before completion (#2240, PR #2243).
• Bounded provider-batch input checks by per-key admission and global capacity, and released reservations when requests fail or are canceled (#2240, PR #2243).
• Enforced model access on Anthropic requests and model/backend access on provider batches (PRs #2200, #2202).
• Prevented callers from overriding Router-owned Anthropic user-profile attribution (PRs #2189, #2204).
• Restored platform CA trust for outbound WebSocket clients (PR #2201).
• Honored per-model include_usage, carried assistant reasoning to Dynamo, and truncated streamed error bodies at character boundaries (PR #2211).
• Corrected Dynamo idle-worker handling, equal-cost rotation, aggregated-fleet scoring, subscription restart load, and prefill/decode worker pairing (PRs #2218, #2219, #2225, #2232).
• Fixed character-boundary masking of rate-limit identity IDs and synchronized the idle log-flush regression test (PRs #2215, #2207).

Changed¶

• Raised the Rust source-build minimum to 1.97 and aligned the vendored Hub protocol minimum (PRs #2199, #2210).
• Updated Rust and WebUI dependencies, including the ctutils fix for RUSTSEC-2026-0385 (PRs #2223, #2230, #2231).

CI¶

• Cleared stale Windows signing staging directories, reserved space for Redis shared-state tests, moved test workflows to dedicated runner pools, and skipped pull-request workflows on draft PRs (PRs #2190, #2214, #2226, #2227).
• Updated astral-sh/setup-uv to 10.3.2 (PR #2229).

Compatibility¶

• Earlier Router releases cannot restore a persisted key table containing Anthropic user profiles. Plan downgrade recovery before enabling Hub-synced profiles (PR #2221).
• Cost-center enforcement remains asynchronous across replicas. Concurrent work can exceed a limit before the next authoritative budget update.

v1.35.1 - 2026-10-22¶

Fixed¶

• List llama.cpp llama-server and mlxcel-server models again (#2181, PR #2182). Both servers answer GET /v1/models with an Ollama-style models array next to the OpenAI data array, and v1.35.0 rejected every such body as having a duplicate data field, so their models disappeared from the aggregated /v1/models. Model discovery now reads data whenever it is present and skips models; the System One TypeSafe models[].name catalog is read only when data is absent.
• Stop reporting a timeouts.health_check conflict when the deprecated block is not written (#2179, PR #2180). An omitted block loaded as 30s/5s defaults, so health_checks with non-default intervals or timeouts logged a false conflict warning. The block is now absent unless configured and is omitted from serialized configuration; a written block keeps its validation and conflict warning.

CI¶

• Fix the Windows x86_64 release build that failed for v1.35.0 (PR #2178). The ARM64 signing job left a sparse checkout at the root of the workspace it shares with the x86_64 build; it now checks its script out into signing-checkout/, and scripts/check_release_signing.py rejects a signer sparse checkout without a dedicated path.
• Install protoc through a local composite action (.github/actions/setup-protoc) that downloads and caches the pinned 36.2 release, replacing arduino/setup-protoc, which targets the deprecated node20 runtime (PR #2177).

v1.35.0 - 2026-10-21¶

Added¶

• Automatic Gemini Native API routing and bidirectional transformation for exclusive features (#85, #163, PRs #2161, #2162). When routing to a type: gemini backend across /v1/chat/completions, /v1/responses, /v1/images/generations, and /v1/images/edits, requests that use Google Search grounding (tools with google_search or web_search, web_search_options, or google_search: true on image endpoints), Python Code Execution (tools with code_execution or code_interpreter), image output modalities (modalities: ["text", "image"], image_config, or a Gemini image generation model), multi-turn assistant images history, extra_body.google native options (grounding, response_modalities, safety_settings, media_resolution, cached_content), or gemini_context_cache.prefer_native_api: true automatically dispatch to Gemini's native /v1beta/models/{model}:generateContent and :streamGenerateContent?alt=sse endpoints using x-goog-api-key (or OAuth / service-account Bearer auth). Responses normalize thoughts into reasoning_content, executableCode and codeExecutionResult into code_execution, grounding links into annotations (url_citation, restricted to http(s) schemes and capped at 128 entries) and grounding_metadata, and generated images into choices[].message.images while preserving round-trip thoughtSignature metadata on tool calls and assistant turns. Adds nano-banana-2 (gemini-3.1-flash-image, up to 4K) and nano-banana-2-lite (gemini-3.1-flash-lite-image, 1K) aliases and SSRF validation (UrlValidator::strict_without_dns) on remote image URLs before native inline fetching.
• Catalog System One decision models in model-metadata.yaml and model discovery (#2163, PR #2166). Adds source-verified catalog entries and aliases for TypeSafe jev-1.13.0 (jev-latest) and jev-1.14.0 (jev-preview), Azure AI Foundry microsoft-decision-1, laya, kev-0.8b, kev-4b, julia-1, lev, openjev, clef, pplx-decider-v1.2, bespoke-nimble-9b-v3, decision-2.0-{eos-0.8b,kai-0.6b,lux-9b,nox-4b,sol-2b,vega-27b}, d3 and d3-{edge,lite,nano,mini,flash}, Liquid AI d1-3b and d1-omni-600m, and dual-capability granite-switch-4.1-{8b,30b}. Model discovery parses TypeSafe GET /v1/models objects and llama.cpp architecture.output_modalities to advertise decisions in model methods, requires an explicit chat capability before multimodal image or audio tags imply text generation, and redacts URL query strings and userinfo in Gemini image fetch warning logs.
• Provider profiles, multimodal media support, and token accounting for System One decision targets (#2164, PR #2168). Extends the shared SystemOneTargetConfig (decisions.targets[], smart_routing.classifier.system_one, guardrails.providers[].systemone, safeguards.action_safety.systemone, fallback.quality_check.target, and usage.quality_judgment.target) with protocol (generic, typesafe, foundry, llama_cpp), auth (auto, bearer, api_key, entra), model_family (generic, openjev, lfm2_d1, lfm2_d1_omni, vllm_d3), typesafe_images, typesafe_long_context, max_questions (1..=256, default 64), and max_media_bytes (1..=16777216, default 4 MiB). Supports structured, null, and omitted question fields on non-generic profiles, TypeSafe JSON assets and box localization questions on jev-1.14.0 (jev-preview), Azure AI Foundry /providers/microsoft/v1/systemone endpoint normalization and TLS/credential validation, and bounded inline files/images data URLs and state: null on supported llama_cpp families, while preserving upstream-reported output_tokens in usage records independently of $0 output pricing.
• Integrate System One provider profiles into the smart-routing classifier (#2165, PRs #2169, #2170). smart_routing.classifier.system_one now dispatches through the profile-aware System One client across generic, typesafe, foundry, and llama_cpp targets, excludes decision-only models (decisions without chat) from chat candidate pools (reported as excluded_incompatible_models on GET /admin/smart-routing/status), keys classifier cache entries on the resolved target profile, model, question battery, and backend topology generation (cache_target_identity), records malformed HTTP-2xx decision payloads as upstream errors, and preserves successful token and cost accounting when a valid decision response omits confidence and falls back to the rule or LLM classifier.
• Source-aware model metadata catalog updates and packaged model-metadata.d vendor fragments (#2158, PR #2174). Metadata documents now accept an optional top-level source_url validated as an https:// URL without userinfo, query strings, fragments, or private/loopback/link-local literals. continuum-router metadata download (alias metadata update) can refresh a single self-sourced file from its embedded source_url or update every source-bearing file in model-metadata.d/ via --all / --dir <DIR> while preserving source-less operator files, enforcing the 64-file / 8 MiB catalog limits before writing, and rejecting symlinks on write targets. Ships 19 vendor and developer catalog fragments under model-metadata.d/ (10-alibaba.yaml through 28-zhipu-ai.yaml, 323 models including six Upstage Solar models) in release archives, Debian packages, and Docker images alongside model-metadata.yaml.

Changed¶

• Narrow Anthropic's default health check accept_status from [200, 400, 401, 429] to [200, 400] so authentication failures (401) and rate-limited probes (429) no longer mark an Anthropic backend healthy by default (#2175, PR #2176). Operators who want the previous behavior can set backends[].health_check.accept_status explicitly.
• Overhaul the English and Korean documentation sets (docs/en/, docs/ko/) for conciseness, structural parity, and direct technical prose, and establish docs/manual_writing_rules.md (PRs #2167, #2172, #2173).

Fixed¶

• Bound provider health check probe traffic and fallback overhead (#2175, PR #2176). Default health probes now request small single-model or lightweight endpoints (/v1/models/gpt-4o-mini on OpenAI, /v1beta/models/gemini-2.5-flash on Gemini, /v1/models?limit=1 on Anthropic, /health on vLLM, SGLang, TensorRT-LLM, llama.cpp, mlxcel, Azure, generic, and Continuum Router, and /api/version on Ollama) instead of downloading full model catalogs on every health tick. Catalog fallback probes (/v1/models, /v1beta/models, /api/v1/models, /api/tags, /) run only when the primary probe returns HTTP 404 or 405, cache positive fallback results for 5 minutes per backend and check context, coalesce concurrent fallback probes without skewing latency or threshold counters, enable automatic gzip decompression on the dedicated probe and model-discovery HTTP client while keeping identity transfer on proxied inference and SSE streams, and bound Unix socket health probes to 64 KiB of response headers and the configured timeout across connect, request write, header read, and body drain.

Security¶

• Sign Windows x64 and ARM64 release binaries, enforce fail-closed signature verification in CI, and upgrade TLS root certificates (#520, PR #2171). .github/workflows/release.yml now signs continuum-router.exe, continuum-kv-listener.exe, and continuum-event-broker.exe for both x86_64-pc-windows-msvc and aarch64-pc-windows-msvc with SSL.com KMS-backed Authenticode (signtool.exe sign + signtool.exe verify /pa /tw), hardens macOS Developer ID signing, notarization, and Homebrew OpenSSL discovery, and guards the release workflow with scripts/check_release_signing.py in CI. Upgrades rustls-webpki to 0.103.16 and removes vulnerable webpki-roots 0.26.11 from the dependency graph by switching outbound WebSocket TLS (tokio-tungstenite) to bundled Mozilla root certificates (webpki-roots 1.0.10) with an explicit ring crypto provider.

v1.34.0 - 2026-10-20¶

Added¶

• Per-operation rate limiting for the Files API (#114, PR #2143). A new optional files.rate_limiting block (enabled, default false; uploads_per_minute, default 10; downloads_per_minute, default 100; operations_per_minute, default 1000; nonzero required when enabled) enforces per-caller token buckets across files_upload (POST /v1/files), files_download (GET /v1/files/{id}/content), and files_operations (GET /v1/files, GET /v1/files/{id}, HEAD /v1/files/{id}, and DELETE /v1/files/{id}). Callers are keyed by SHA-256 hashed API key, authenticated user ID, or client IP, and requests above the limit answer 429 Too Many Requests with Retry-After and x-ratelimit-* headers.
• Refresh the built-in model metadata catalog (model-metadata.yaml) with late-September and October 2026 releases (PR #2160): Google gemini-4-argon, Meta muse-wonder-1.0 and muse-spark-open-816b-a42b, Mistral mistral-large-4, Upstage solar-pro-4 and solar-mini-4, NVIDIA nemotron-3.5-super / nemotron-3.5-ultra / nemotron-labs-dory1.0-4b, Liquid AI lfm2.5-* and d1-*, Jina AI jina-ocr-v1, DeepSeek deepseek-v4.1-flash-1017, Alibaba qwen3.8-* and qwen-image-2.1-turbo, Tencent hy-memory-30b-a3b, Meituan longcat-*, Ant Group ling-3.*, Shanghai AI Lab atria-dawn-preview and intern-s2-*, and StepFun step-5-preview.

Changed¶

• Coalesce only in-flight deterministic requests in the request deduplicator (#2157, PRs #2156, #2159). Request deduplication now coalesces identical concurrent requests only while the leader is in flight and removes the 30-second post-settle cache of successes and durable errors (cached post-settle responses belong to response_cache). Chat and legacy completions (/v1/chat/completions and /v1/completions) are eligible only when the request carries an explicit numeric temperature: 0 (0 or 0.0), and /v1/images/generations is never coalesced. A waiter joined to an in-flight leader is metered as a local cache hit (backend: not_dispatched, cache_hit_type: exact, the leader's prompt tokens, zero completion tokens) and is not charged a second time against token rate limits or Hub budgets. Library breaking changes: removed Admission::Cached, DeduplicationEntry::{with_success, with_error, response, response_backend, error, in_flight}, DeduplicationManager::{cache_success_with_backend, cache_error}, and DeduplicationStats::{total_entries, cached_responses, cached_errors} (replaced by waiting); FileService::list_files_metadata (#133, PR #2140) now returns FileListPage instead of Vec<FileMetadata>, and the unused BatchRequest and BatchOptions types (#2135, PR #2145) are removed from core::models.

Fixed¶

• Deliver the leader's outcome to deduplicated waiters within the request budget and count fallback-hop dial waits (#2133, #2149, PRs #2141, #2151). DeduplicationManager::admit uses a single atomic DashMap::entry check (Lead or Join) with a tokio::sync::watch channel and a drop guard that clears the in-flight marker if the leader is cancelled, and the waiter budget sums each attempt's request timeout plus the retry backoff ceiling (and timeouts.connection on fallback hops) with saturating deadline arithmetic.
• Key deduplicated requests on a closed header allow-list and canonical JSON (#97, #2150, PRs #2142, #2154). The header portion of the deduplication key now hashes only a closed allow-list of credential headers (authorization, x-api-key, x-goog-api-key), provider behavior headers (anthropic-version, anthropic-beta, openai-beta, openai-organization, openai-project), router control headers (x-backend, x-continuum-debug), and headers referenced by the active config snapshot's routing.rules[].when.header or sticky_key, in sorted length-prefixed form, so per-request tracing and SDK telemetry headers (X-Trace-ID, x-request-id, x-stainless-*, user-agent) no longer split identical requests into separate upstream calls. JSON objects in deduplication and response-cache keys are canonicalized with sorted keys and -0.0 normalized to 0.0.
• Resolve Anthropic runtime registration, URL, model matching, cache, and health check bugs (#2139). Prevent /v1/v1/messages path duplication when an Anthropic backend url ends with /v1, populate the default provider URL in Admin backend create, update, and probe requests when url is omitted, match dateless allowed_models entries and model aliases against date-suffixed live Anthropic model IDs (preferring the newest date), avoid overwriting a cached model list with an empty discovery response on revalidation, and send x-api-key and anthropic-version: 2023-06-01 on Anthropic health checks.
• Filter file listings by owner and purpose before applying after and limit (#133, PR #2140). GET /v1/files and GET /admin/files now filter by purpose and caller ownership before slicing the page (limit in 1..=10000, default 10000), compute has_more from the filtered set, and treat an after cursor that is missing or belongs to another owner as an empty page so callers cannot probe other owners' file IDs.
• Accept multiple edit images on POST /v1/images/edits, resolve model aliases before the edit check, and fix Nano Banana documentation (#2134, PR #2144). /v1/images/edits now accepts up to 16 image or image[] multipart fields with per-model count limits (16 for gpt-image-*, 14 for nano-banana, nano-banana-pro, and Gemini image models, and 1 for dall-e-2 and /v1/images/variations), forwards multiple images to OpenAI-compatible backends as image[] parts and to Gemini as multiple inlineData parts, and resolves model_aliases before checking SUPPORTED_EDIT_MODELS.
• Reject router-local file IDs on POST /v1/batches (#2135, PR #2145). When files.enabled is on and input_file_id names a file stored in the router's local Files API, POST /v1/batches now fails fast with 400 Bad Request (invalid_request_error, code router_local_batch_input_file) before routing deferral or batch admission instead of forwarding a router-local ID to an upstream provider.
• Capture live vLLM 0.32.0 and SGLang 0.5.22 tokenizer fixtures and refuse options SGLang ignores or misrenders (#2020, PR #2153). SGLang /v1/tokenize silently ignored return_token_strs, add_generation_prompt, add_special_tokens, continue_final_message, and chat_template, and concatenated messages as raw role-colon lines instead of applying the model's chat template when tools were absent; the router now refuses those unsupported options on SGLang with 400 tokenize_option_unsupported (or routes to a capable sibling backend) and never synthesizes token strings the engine did not send. Added an end-to-end SGLang Prometheus metrics scraper test against the captured 0.5.22 exposition fixture (#1486, PR #2148).

CI¶

• Guard release binary feature union parity across .github/workflows/release.yml, .github/workflows/ci.yml, scripts/local-ci.sh, and Cargo.toml (#1311, PR #2152). scripts/assert-release-feature-parity.sh and its mutation test suite run in the shell-syntax job.
• Bump dtolnay/rust-toolchain in /.github/actions/rust-toolchain (PR #2147).

Dependencies¶

• Update lru 0.16.3 to 0.19.0, tokio 1.53.2 to 1.54.0, and aws-smithy-types 1.6.1 to 1.9.0 across the root and standalone crate lockfiles (PR #2155).
• Bump the minor-and-patch group with 2 updates (PR #2146): arc-swap 1.9.2 to 1.9.3 and tokio 1.53.2 to 1.54.0.

v1.33.0 - 2026-10-15¶

Added¶

• proxy.internal_streaming meters a buffered client request that is cancelled mid-generation (#2043, epic #1624). With the new proxy section's internal_streaming (default false) or a per-backend backends[].internal_streaming override on, an eligible buffered /v1/chat/completions or /v1/completions request is sent to an OpenAI-wire backend (vllm, sglang, tensorrt-llm, llamacpp, mlxcel, ollama, lmstudio, openai, generic, continuumrouter, over HTTP(S)) with stream: true and stream_options.include_usage: true (plus continuous_usage_stats on vllm, through the #2041 writer, so a cancel there records the engine's own count as engine_continuous), read as a stream, and answered with the ordinary buffered body the router assembles from it (content, reasoning_content, refusal, tool_calls with content: null when only tool calls streamed, per-choice finish_reason, and the engine's usage). A client that disconnects mid-generation is now recorded once as outcome: client_cancelled with the router's count of the streamed deltas, stream: false, and http_status: 200, where it used to leave no record; a completion without engine usage records usage_missing; a stream that fails after it generated output records upstream_error (or timeout_mid_stream) with the partial count, answers 502 (or 504), and is final, with no retry and no fallback hop, while one that fails before any output retries and falls back like a failed buffered read. Only the client chat and completions routes opt in, so the /v1/decisions emulation, the Anthropic Messages and Responses ingresses, embeddings, images, probes, and router-initiated calls stay buffered byte for byte, as do requests with n > 1, log probabilities, a json_schema response format, legacy functions, audio output, or web_search_options. true is a load error on anthropic, gemini, bedrock, azure, and unix: backends; both fields reload immediately; the Admin API accepts the override and the Hub local export reports it as unsupported. The streaming cache replay now uses the same assembler, so a cached stream replayed to a buffered client also carries its tool calls and reasoning. See "Internal streaming for buffered requests" in docs/en/configuration/index.md.
• Usage reconciliation against engine generation counters (#2046, epic #1624). A new optional usage.reconciliation block (enabled, window 10s to 24h, default 5m; max_drift in (0, 1], default 0.05; history 1 to 1000, default 24) compares, per backend and window, the output tokens the usage ledger recorded with the delta of the engine's cumulative generation counter that engine_stats already scrapes (vLLM, SGLang, llama.cpp, and mlxcel report one; other types report no_engine_data). Each window closes as within_tolerance, drift (one warning), no_engine_data, or reset; the first engine value only sets the baseline, and a window without a fresh scrape carries its ledger tokens forward. Results are served at GET /admin/usage/reconciliation and as usage_reconciliation_drift_ratio, usage_reconciliation_ledger_tokens_total, usage_reconciliation_engine_tokens_total, usage_reconciliation_windows_total{result}, and usage_reconciliation_dropped_samples_total. backends[].reconciliation: false opts a backend out (for an engine shared with other traffic). The reconciler is always compiled, independent of control-plane, and taps the same two recorder seams under the same rule as the usage export. Out-of-range values are load errors (reconciliation_window_out_of_range, reconciliation_max_drift_out_of_range, reconciliation_history_out_of_range), and config validate warns when the engine side cannot be read or when window is shorter than twice a reconciled backend's effective engine_stats.interval (its per-backend override, else the global one). A backend outside the reconciled set never takes a slot in the 8192-sample queue, so a busy opted-out engine cannot drop a reconciled backend's samples. tests/usage_reconciliation_load_test.rs is the epic's mixed-load evidence: completed, cancelled, mid-stream-fallback, and chunk-timeout streams agree with two mock engines within 1%.
• Engine-accurate partial usage for streams cut before their final usage block (#2041, epic #1624 unit D). Every streaming dispatch over an OpenAI-wire chat completion (each /v1/chat/completions attempt and fallback hop, the native Gemini stream, the /anthropic/v1/messages conversion, and the /v1/responses conversion) now shapes stream_options through one helper that merges the client's object instead of replacing it: include_usage is forced on, a vllm backend is also asked for continuous_usage_stats (a cumulative usage on every chunk, recorded as engine_continuous when the client cancels or a timeout cuts the stream) unless the request asks for more than one choice (n other than 1), since vLLM's per-chunk usage counts only one choice, and every other backend type, and every multi-choice request, has a client-sent continuous_usage_stats removed. SGLang and TensorRT-LLM carry the option in their upstream source but are not injected without a captured stream; the engine table and the capture procedure are in the architecture document and tests/fixtures/continuous_usage_stats/README.md. A new optional usage.tokenize_correction block (enabled false, timeout_ms 2000 in 100-30000, max_concurrent 4 in 1-256) recounts a stream recorded with the router's own delta count: after the client stream ended, the text the router received is posted once to the serving engine's tokenizer route (vllm, sglang, llamacpp, mlxcel, continuumrouter, with add_special_tokens: false) and the record is emitted once, at most timeout_ms later, as engine_tokenize with its outcome unchanged; the count is a lower bound, since template control tokens between segments are not in the text. The engine call never touches the backend's circuit and carries the hop count captured at dispatch. usage_tokenize_corrections_total{result} counts each considered stream once (applied, timeout, unsupported, skipped_truncated, shed, error); an engine answer over 1 MiB, or a count of zero for non-empty text or above the text's UTF-8 byte length, is not applied and counts error. Superseded mid-stream fallback attempts are recorded at once and never corrected, and a correction still in flight when a graceful shutdown starts emits after the control-plane agent's final push and is lost (a known limitation, bounded by timeout_ms). The /anthropic/v1/messages conversion no longer emits a second, identical message_delta for the trailing usage chunk of a per-chunk usage stream.
• Tamper evidence for the usage ledger (#2047, epic #1624). Every usage batch the control-plane agent builds now carries three additive UsageBatch fields: batch_hash (SHA-256 over prev_batch_hash, router_id, batch_id, sent_at_ms, chain_gap, and each record's canonical JSON), prev_batch_hash (the last batch built before it, delivered or not), and chain_gap (built batches lost since, for example after a 413 or a spool eviction). sent_at_ms is now fixed when the batch forms, so a retried batch is byte-identical. The chain head persists in <state_file>.usage-chain.json (0600, bound to the enrolled router_id); a failed write or rename removes it, so a crash before the next write restarts the chain instead of forking it. The new usage.ledger_export block (path, max_bytes, rotate) appends every built batch to an owner-only JSONL file that is never truncated, rotating to <path>.<sent_at_ms> with an anchor line; a file started after the chain began opens with an anchor too, so every file the router writes verifies on its own, and scripts/verify-usage-ledger.py verifies a file with the Python standard library (--prev joins it to the file before). An empty path, one with a control character, a max_bytes below 1 MiB, and a path that is control_plane.state_file or a file named after it (ledger_export_path_conflicts_with_state_file, which the writer also checks before every open) are load errors. The block reloads at the next push tick without pausing the usage loop: the new writer waits on the blocking pool for the old one (at most 5 seconds, after which the old one drops what it has not started), so two writers never append to one file. New metrics: usage_batches_hashed_total, usage_chain_gaps_total, usage_ledger_export_failures_total{target}, and usage_ledger_export_bytes. The protocol is synchronized with Hub PR #1397 (continuum-hub#1393); no local-ahead protocol files remain.
• Router-initiated LLM calls are metered (#2051, epic #1624). Every call the router makes on a client request's behalf produces one usage record per dispatch, each retry included, tagged with its purpose: guardrail (the LLM guardrail providers and the systemone provider, one record per chunked window check), classifier (the smart-routing LLM and System One classifiers, including a parse-failure retry), action_safety (the action-safety LLM and System One evaluators), and quality_judgment (the usage-quality judgment and the fallback quality check). The record carries the triggering request's key_id, request_id, and its own attempt_seq with no parent_attempt; a pinned backend is named by its backends[] name and an inline endpoint by its host[:port]. A reply without usage records usage_missing with char_estimate counts, a non-success status upstream_error with that status, a call abandoned after its dispatch started (the guardrail service's timeout_ms, a dropped client request) timeout_first_byte, and a call refused before dispatch nothing. The router's local key budget is never charged for these calls (they are attributed through PolicyStore::hub_identity_for_token, which charges nothing), and /v1/decisions still records only the client. Sequence 1 is reserved for the client's first attempt, internal calls are not counted in attempts_total, and GET /admin/stats gains a by_purpose split, whose client entry counts client requests and whose other entries count internal calls; their tokens are in overall and llm_tokens_total under the client key, and the windowed view keeps internal calls in their own bounded ring so they never evict client rows from the load summary or the percentiles. An internal record never takes the last tenth of the control-plane usage channel, so during a slow hub push it is dropped first (counted on usage_records_dropped_total{reason="channel_full"}) rather than crowding out client records, and a mid-stream fallback hop (#2039) takes the request's next sequence from the same allocator, so it never reuses a guard call's attempt_seq and may skip past its parent. Billing-visible on the hub. See "Router-initiated calls" in docs/en/architecture.md.
• Admin usage export by period, model, backend_id, outcome, and purpose (#2045, epic #1624). GET /admin/usage/export returns the router's per-attempt usage records as JSON or CSV, raw (sorted by (completed_at_ms, request_id, attempt_seq), paged by limit and a cursor that CSV carries in X-Next-Cursor) or grouped by period:1h/period:1d, model, backend_id, outcome, purpose, and key_id with requests, attempts, and token sums, filterable by outcome, purpose, model, and backend_id. CSV uses CRLF line ends and minimal RFC 4180 quoting, and a field a spreadsheet would evaluate as a formula (leading =, +, -, @, tab, or CR) or one that already starts with ' is written with a leading '. share_of adds the FAQ 12 output-token share, excluding local cache replays and counting every outcome and, absent a purpose filter, every purpose. The records come from a bounded ring behind admin.stats.usage_export (enabled, default false and also needing admin.stats.enabled; retention, default 7d; max_records, default 1,000,000 within 1,000 to 10,000,000; a bad value is a load error; every field reloads immediately; while either switch is off the endpoint answers 404 usage_export_disabled), appended for every attempt that reached a backend, every relayed stream, and every local cache replay (backend: local-cache, cache_hit_type set, no output), never for a request that reached no backend. They ride the stats snapshot when persistence is on, at about 300 bytes per record, and usage_export_records_evicted_total{reason} counts what the ring evicts. The export is per router; on a fleet, sum every router's export or use the hub. See "Usage export" in docs/en/admin-api.md.
• Usage ledger durability across hub outages and restarts (#2040, epic #1624). A write-ahead attempt log, <state_file>.usage-wal.jsonl (0600), records a start line for every buffered and streaming /v1/chat/completions backend dispatch (one per retry attempt and fallback hop) and an end line when it finishes; after a kill or crash the next boot pushes each unfinished attempt as one outcome: router_restart record with the router's input estimate, output_tokens: 0, token_source: none, and estimated: true, sequenced and charged like any record. A bounded disk spool, <state_file>.usage-spool/ (0700), replaces the push buffer's drop-oldest: an overflowing buffer writes its oldest batch there, the push loop drains it first under the original batch_id and router_usage_seq, and a shutdown spools the unacked batch and the buffer for the next process. Two hot-reloadable keys bound them, control_plane.usage_wal_max_bytes (default 64 MiB, 1 MiB to 1 GiB) and control_plane.usage_spool_max_bytes (default 256 MiB, 0 keeps drop-oldest, otherwise 1 MiB to 64 GiB). Every remaining loss is counted in usage_records_dropped_total{reason} (channel_full, buffer_overflow, spool_overflow, batch_rejected_413, ack_oversized, wal_dropped), skipped corrupt files in usage_durability_corrupt_total{kind}, and the spool in usage_spool_records and usage_spool_bytes; spooled records count toward the heartbeat backlog and the control-plane status queue depth. The /v1/responses, /anthropic/v1/messages, and passthrough ingress do not write WAL lines yet. A WAL or spool file that is a symlink, not a regular file, or not owner-only is skipped (a spool file is deleted and counted), and a spool batch written under another router enrollment is discarded and counted; graceful shutdown waits up to one hub request timeout plus 5 seconds for the agent to flush the spool and settle the WAL. See "Durability across outages and restarts" in docs/en/architecture.md.
• Model artifact and draft-model identity on model metadata and usage records (#2048, epic #1624). ModelMetadata gains an operator-declared artifact (digest as sha256: or blake3: plus 64 hex, canonicalized to lowercase; registry_version; revision; source, a URI whose userinfo is refused) and draft_model (name, optional artifact), resolved per backend and model one key at a time: a backend's model_configs entry wins over model-metadata.yaml and its drop-ins and replaces the shared artifact whole, and the built-in catalog is never consulted. Every usage record that reached a backend carries the additive model_digest and draft_model (a fallback hop carries the serving backend's identity, a batch completion the one captured at submit, a cache replay neither), now synchronized with the Hub protocol after continuum-hub#1385; PROTOCOL_VERSION stays 0. draft_model names the draft and counts nothing: accepted tokens belong to the target model. GET /v1/models/{model} shows the first serving backend's identity, GET /admin/models lists every serving backend's under artifacts. A malformed identity is a load error in model_configs, warned and dropped in the metadata layer (listed by metadata show --resolved), and refused by metadata download; config validate warns on a vllm, sglang, or tensorrt-llm draft without digests and on one model id declared with different digests. See "Model artifact identity" in docs/en/configuration/index.md.
• Attempt-scoped usage ledger schema (#2053, epic #1624). Every control-plane usage record now carries fourteen additive fields: outcome (how the attempt ended: completed, length, and the reserved terminating causes), token_source (how the output count was obtained, engine_final today), estimated, stream, http_status (the upstream status of the attempt), request_id (a router-minted req_ id shared by every attempt of a client request, never the client's x-request-id), attempt_seq and parent_attempt, purpose (client today), reasoning_tokens (read from the Chat Completions and Responses usage detail objects into TokenUsage), and the UTF-8 input_chars, input_bytes, output_chars, output_bytes of the text sent and received. The vendored continuum-protocol crate is re-synced to the hub revision that defines them (continuum-hub#1380); PROTOCOL_VERSION stays 0. Every producer stamps them from day one through a request-scoped ledger installed on the API route group, and the router applies the hub's drop rules before sending. Locally, GET /admin/stats gains by_outcome (a stats snapshot written before the upgrade restores its windowed rows as completed, while the all-time by_outcome split starts empty and counts only requests recorded after the upgrade), llm_tokens_total gains the outcome label, and usage_records_total{outcome, token_source, purpose} counts every record. See "Usage record outcome and attempt identity" in docs/en/architecture.md.
• Router-observed output token counting and one stream-usage finalizer (#2052, epic #1624 unit C1). A streamed completion whose backend reports no usage block (an engine without stream_options.include_usage, a proxy that strips it, a /v1/completions engine, a lost terminal frame) is now charged instead of recorded as nothing: every streaming relay owns a StreamUsageFinalizer from its first event and records exactly one usage event at its normal end, with outcome=usage_missing, token_source=router_delta_count, estimated=true, the router's own count of the streamed content, reasoning, and tool-argument deltas (one per chunk, the engine's granularity on vLLM-class engines) as the output, the request's estimated input (or the engine's prompt when it reported only that), and the UTF-8 output_chars/output_bytes. The chat pipeline tap is latest-wins and finalizes once (client_cancelled when the client leaves before the end), min_accumulated_tokens for mid-stream fallback now counts delta events rather than characters divided by four and a reasoning-only partial never continues, and the new usage_missing_total{backend} counter shows the rate of router-estimated records. See "Router-observed output counting and the stream-usage finalizer" in docs/en/architecture.md.
• Realtime usage metering (#2044, epic #1624). GET /v1/realtime sessions now meter the usage their backend reports: every server response.done the router relays becomes one usage record through the shared stats recorder, reaching /admin/stats, the per-key llm_tokens_total counter, and the control-plane usage record with stream: true, http_status: 101, the session's request_id, a per-session attempt_seq, and an outcome mapped from response.status (incomplete with max_output_tokens is length, cancelled is client_cancelled, failed is upstream_error). Records are charged to the model the session was dispatched for, never to a response.model or session.model on the wire, which the backend authors and the router does not validate. A failed or cancelled response with no usage is still recorded, with zero tokens and token_source: none. The relayed bytes are unchanged: only a copy of each backend text frame of at most 256 KiB is parsed, after it was sent, and binary and client frames are never inspected. The new realtime_usage_events_total{backend, result} counter reports recorded, oversized, unparsed, and session_closed_in_flight (a response the session ended on before its response.done, which records nothing). See "Usage metering" in docs/en/configuration/realtime.md.
• Hub-declared guardrail provider rows that reference a served backend (#2082, PR #2128). A control-plane router now advertises guardrail_provider_rows_v1 and composes the Hub policy's provider_rows (llamaguard, llm_judge, and a backend:-referenced openai_moderation) after its local rows in declared order. A local row of the same name wins and is reported with overrides_hub: true; Hub rows are reported with source: "hub" in the guardrail inventory and GET /admin/guardrails, turn guardrails on for a router that has none locally, and are never written to disk by PATCH /admin/guardrails. A row whose backend is missing or disabled in the effective configuration (Hub-synced backends included, internal: true accepted) is refused with guardrail_backend_missing, guardrail_backend_disabled, or guardrail_providers_unavailable. The vendored continuum-protocol crate moves to Hub revision 47e111b0.
• Select the guardrail route from the Hub tier snapshot (#2096, PR #2129). The router advertises guardrail_route_v1 and reads TierLimits.guardrail_route, so keys on different tiers receive different guardrail policies for the same model without a config.yaml edit at key issuance. The effective policy layers global defaults, then guardrails.routes[<tier route>], then guardrails.routes[<model>] (the stricter mode wins, stages are unioned, lower per-category thresholds win), and a Hub tenant guardrail policy still composes strictest-wins over all local routes. A tier route that names no configured route falls back to the global policy and counts guardrail_route_unknown_total. The tier route is part of the response-cache and tool-verdict-cache scopes, survives the persisted key table, is shown on GET /admin/control-plane/policy, GET /admin/guardrails, and POST /admin/guardrails/test, and labels guardrail_blocks_total and guardrail_verdicts_total with a bounded route (16 values, then other).
• Sanitize-then-inspect guardrail staging (#2100, PR #2127). Guardrail providers take phase: sanitize | inspect (default inspect); only transform-capable providers (pii, bedrock_guardrail/aws_bedrock) may use sanitize, checked at startup and hot reload. Input documents, non-streaming and buffer_full output, chunked stream windows, and dry runs are evaluated in the order deny, allow, sanitize, mask, inspect on the masked view, final verification, so an inspect provider never sees text a sanitize provider masked. A failing sanitize provider follows on_error (fail_closed blocks before any inspect provider runs), and in monitor mode and dry runs a sanitize block or masking failure stops before unmasked text reaches a remote inspect provider. Measured staging overhead is about 0.2 ms at p50.
• Per-route guardrail streaming mode and window settings (#2099, PR #2112). guardrails.routes[] entries accept streaming_mode, streaming_chunk_size, streaming_context_size, and streaming_stream_first, inheriting the global values when unset, so a latency-sensitive route can run chunked while a strict route on the same router runs buffer_full. streaming_mode: passthrough on a route is refused unless that route's effective mode is monitor. GET /admin/guardrails shows the resolved values per route; Hub composition leaves them router-local.
• Korean PII identifiers and custom regex entities in the built-in pii provider (#2095, PR #2115). New kr_rrn, kr_frn, and kr_brn scanners for resident, foreigner, and business registration numbers run ahead of credit_card and phone, so those numbers are no longer reported as cards or phone numbers; an unhyphenated 13-digit number needs a nearby context word. options.custom_entities (name, pattern, optional context_words and context_window, at most 64 entities and 1024 bytes per pattern) defines operator entities whose action comes from actions[name] or default_action. GET /admin/guardrails lists the effective entities and PUT /admin/guardrails/providers/{name} can update options.
• Enforce Hub-delivered request priority on vLLM backends (#2093, PR #2126). A Hub tier can carry TierLimits.request_priority (-100 to 100, lower runs first under vLLM --scheduling-policy priority), advertised as request_priority_v1 and persisted in the key table sidecar. On /v1/chat/completions and /v1/completions to a vllm backend the router injects or clamps the top-level priority field and X-Vllm-Priority after request_extensions are applied, and clamps an existing /v1/embeddings priority, so neither the client nor a backend default can schedule above the tier. A response that had priority applied carries x-continuum-request-priority; a tiered request routed to a non-vLLM backend counts continuum_request_priority_skipped_total{reason="backend_type"}.
• Priority-labelled request and TTFT series (#2094, PR #2130). continuum_request_priority_requests_total{priority, status_class} and continuum_request_priority_ttft_seconds{priority} compare request rate and router-observed time to first token across Hub-delivered priority classes, even when several tiers share one vllm backend. The priority label is bounded at 16 values (then other), and unmanaged marks a request with no priority in effect.
• OpenAI audio transcription and translation routes (#2102, PR #2120). POST /v1/audio/transcriptions and POST /v1/audio/translations forward the multipart/form-data upload to the selected backend and relay its status, Content-Type, and body unchanged for every response_format. The model form field drives per-model rate limits, the Hub tier model allowlist, and cost-center checks; stream=true answers 400 streaming_not_supported, and an upload is dispatched once without retries. New limits: proxy.audio.max_body_size (default 25 MiB, 413 above it without buffering the body), proxy.audio.max_concurrent_uploads (default 8), timeouts.request.audio (60s first byte, 300s total), and timeouts.limits.max_audio_timeout (default 600s, ceiling 1800s). Both routes require the write scope.
• Audio-seconds metering on usage records and the Hub price table (#2103, PR #2125). Usage records gain an optional audio_seconds and Hub price entries an optional audio_microusd_per_second. A transcription or translation response reports billed duration from usage.seconds (when usage.type is duration) or the verbose_json duration rounded up to a whole second, and its token fields stay 0, so key TPM windows, token budgets, and cost-center token budgets do not move.
• backends[].record_usage_missing (#2101, PR #2117). With this per-backend switch on (default false, alias record_missing_usage, reloads immediately), a 2xx non-streaming response that carries no usable usage block still writes a usage record with outcome: usage_missing, token_source: none, and zero tokens. /v1/embeddings and /v1/rerank now forward request headers to the usage recorder so key, user, and team attribution is kept. The field is accepted by the Admin backend create, update, and probe requests.
• Structured log line for requests rejected before a backend attempt (#2107, PR #2121). Every rejection at the auth, rate_limit, policy, admission, guardrail_input, or routing stage writes one WARN line with event=request_rejected, carrying the trace id, x_request_id, client_ref, request_id when a ledger scope exists, method, route, status, error type and code, limit_type, stage, key_id (never key material), and latency. logging.rejections.max_per_minute (default 600) caps the lines; excess lines are counted in request_rejected_log_suppressed_total and summarized once per window as event=request_rejected_summary.
• Export the authentication, policy, and input-guardrail stages as child spans (#2105, PR #2118). With OTLP export active, router.auth, router.policy, and router.guardrail_input are ordered, non-overlapping children of router.request, each carrying router.stage_outcome (allowed, blocked, or error); the guardrail span also carries router.guardrail.provider. Without export the spans are not created.
• Derive the outbound traceparent from the active span context (#2106, PR #2119). With OTLP export active, the traceparent and tracestate sent to a backend name the router.backend_call span, and a router.request with no inbound traceparent uses the router's own trace id as its OpenTelemetry trace id. The sampled flag (-01 or -00) follows the resolved sampling decision with or without export, and an invalid inbound tracestate is dropped before propagation.
• Persist the configuration history across restarts without secrets (#2098, PR #2113). With admin.history_dir set, every file-edit hot reload writes the raw configuration text with its ${VAR} placeholders intact (never resolved secrets) to history-<version>.json (0600, directory 0700), trimmed by admin.max_history_entries (default 100) and admin.max_history_bytes (default 10 MiB). GET /admin/config/history and POST /admin/config/rollback/{version} merge the persisted entries with in-memory ones; a rollback re-resolves the placeholders. Changes applied through POST /admin/config/apply stay in memory.
• Report a model's accepted reasoning_effort levels on GET /v1/models/extended and GET /v1/models/{model} (#2081, PR #2109). metadata.reasoning_efforts is filled from the built-in OpenAI reasoning families when neither model_configs nor model-metadata.d/ declares it (none is excluded, since rejects_reasoning_effort_none governs it), and the single-model endpoint now also returns reasoning_efforts and rejects_reasoning_effort_none.
• Tokenize chat messages on llama.cpp and mlxcel and through a chained router (#2002, PR #2019). POST /tokenize with messages renders the chat through the engine's POST /apply-template and tokenizes the result under one admission and one 30-second budget. An option the engine would ignore (a per-request chat_template, a non-text content part, an unsupported switch) answers 400 tokenize_option_unsupported naming it, and selection prefers a backend that can honor it. A continuumrouter backend is now a tokenizer type: the body is forwarded with model rewritten, and a downstream 404 or 501 answers 501 tokenization_not_supported unless the code is model_not_found.

Security¶

• Bound router-to-router forwarding loops with a hop counter (#2025). A backend whose URL is the router's own listener, or a cycle of routers (A to B to A), nested one inbound request per hop without bound, each hop holding its request open while it waited on the next, which could exhaust connections and memory on every router in the cycle. The loop was not specific to type: continuumrouter: an untyped (generic) or openai backend pointing at a router looped the same way, and a Hub-managed backend topology could create one. Every request now carries x-continuum-router-hops (absent counts as 0), read before authentication, rate limiting, and the concurrency ceiling: a request that arrives with 8 or more is refused with 508 router_loop_detected before any handler starts (a JSON answer even for stream: true), and every other request is forwarded with the count incremented, through every client-header forwarding path and explicitly on the tokenizer, count_tokens tokenize, image edit, realtime handshake, and /v1/decisions passthrough calls. A non-numeric value answers 400 invalid_router_hops. The 508 is not retried, does not hop a fallback chain, and does not count against the circuit breaker, so a self-loop ends after nine nested requests. The header is reserved for request_extensions.headers (a load error in any spelling), and config validate and the configuration load warn about a backend URL equal to a server.bind_address entry. See "Forwarding Loops and the Hop Limit" in docs/en/configuration/backends.md.
• Carry the router hop count on calls the router makes on a request's behalf (#2032). The hop counter of #2025 covered every path that forwards client headers, but several internal calls built their headers from scratch, so a call whose target was a router that made the same call again restarted the count at every level and nested without bound. These now carry the count of the request they serve: the guard call of an LLM-transport guardrail (self_hosted_classifier, classifier, custom_classifier, by endpoint or backend), every System One decision call (the systemone guardrail, the smart-routing system_one classifier, action_safety with evaluator: systemone, and fallback.quality_check), the control-plane provider batch calls (/v1/batches create, retrieve, list, cancel, and results), and the usage.quality_judgment call, which runs after the reply and sends the count captured from the request it judges, so a judgment cycle between two routers ends within four judgments instead of one client request sustaining paid judgments forever. A guardrail classifier endpoint's 508 is no longer retried or counted against its breaker, and a guard target's 508 now fails closed whatever on_error says: the count starts from a value the client sends, so under fail_open a client could otherwise skip a guardrail served through another router by sending x-continuum-router-hops: 7. A /v1/decisions target's 508 now answers 508 router_loop_detected and never advances the fallback chain; it answered 502 decision_upstream_error, which sent every level above to the chain's emulation hop. The 508 refusal line is rate limited (one WARN per minute, repeats at DEBUG), since the guard runs before authentication and any client could make every request write one.
• Return a fixed error.message on guardrail block responses (#2097, PR #2110). Under block_behavior: error the block response put the verdict's reason into error.message, so a custom_classifier provider's model-written reasoning, which can carry injected text or PII, reached the caller. Every block on /v1/chat/completions, /v1/completions, /v1/responses, and /anthropic/v1/messages, input or output, buffered or mid-stream, now answers This request was blocked by a content-safety policy and cannot be completed.; the status, envelope type and code, and x-guardrail-* headers are unchanged. Clients that matched on provider wording must match on the code.

Changed¶

• Default control_plane.policy.unmatched_key to reject when control_plane.policy.enabled is true (#2092). Once a Hub policy has been applied (or a non-stale key table has been restored from <state_file>.policy-keys.json), a missing credential or a presented API key that matches no synced Hub entry now returns 401 (authentication_error, invalid_api_key) unless the request already carries a locally authenticated AuthContext, so expired or revoked Hub keys that age past the Hub's 30-day retention window and leave the policy envelope cannot fall through to permissive local authentication. Cold start before any policy is applied (policy_sync.source: none) still fails open, and locally configured keys remain unaffected. Note that this rejection also applies to newly created Hub-issued keys during the propagation window until the router applies the policy containing them (default poll 5 s). Operators can measure impact beforehand using control_plane_unmatched_key_total{action} before switching or upgrading. Migration note: deployments that rely on api_keys.mode: permissive admitting unauthenticated or non-Hub keys alongside Hub policy enforcement must set control_plane.policy.unmatched_key: fail_open explicitly; doing so emits a startup WARN log and a continuum-router config validate warning naming the consequence.
• Usage is recorded per backend attempt (#2049, epic #1624). Every dispatch that reached a backend is now one usage record linked to its client request by request_id, attempt_seq, and parent_attempt: a buffered retry, a fallback-chain hop, a pre-stream streaming hop, a quality-check hop, and every turn of a web_search tool loop. This changes what is billed: a failed attempt whose error body reported tokens now charges those tokens; a zero-token failed attempt (a 5xx, a 429, a connect error, a dial timeout) appears as its own record with outcome: upstream_error (or timeout_first_byte), its own http_status, zero output, and an estimated input flagged estimated: true; a quality-check hop is one more attempt of the same request instead of a second request in /admin/stats; and a web_search loop charges every turn, not only the final one. GET /admin/stats gains attempts_total, successful_attempts, and failed_attempts on overall, models[], and backends[] while total_requests keeps counting client requests. A stats row or fallback_attempts_total/fallback_success_total label no longer reads unknown for a dispatched attempt: it names the backend the attempt dialed, and a request that never reached a backend is recorded under not_dispatched. See "Per-attempt records" in docs/en/architecture/model-fallback.md.
• Write the process log in the format and at the level the configuration declares (#2042). A file declaring format: "json", which every generated sample does (Config::default(), the configuration builder, config.yaml.example), now produces one JSON object per line instead of single-line text, so a log pipeline that parsed the text lines must parse JSON or set format: pretty, which keeps the previous lines byte for byte. RUST_LOG is now honored on every path that installs the process subscriber (the server and the config, metadata, auth login, mcp-serve, and migration subcommands) as a full tracing filter directive that overrides CONTINUUM_LOG_LEVEL and logging.level; a value that does not parse is reported on stderr and ignored. Because RUST_LOG used to be inert, a directive that names only continuum_router targets, such as the RUST_LOG=continuum_router=info the systemd unit in the installation guide used to set, now silences the admin_audit and guardrail::audit lines and every dependency's warnings; keep a bare default level (info,continuum_router=debug) or remove the variable, and the router warns at startup when Admin audit logging is enabled but filtered out. The sample docker-compose.yml no longer sets RUST_LOG=info by default (it passes an empty value, which counts as unset), so the logging.level of a mounted file applies.

Fixed¶

• Close the post-merge gaps in the usage ledger (PR #2073, epic #1624). Finalized Chat and Responses streams and intermediate attempts now count in aggregate and windowed token statistics without adding requests, Responses streams are exported without per-key statistics, an internal stream cancelled after the response head but before the first event is recorded, and internally streamed SSE responses are bounded at 16 MiB with partial usage kept and no retry or fallback after the refusal.
• Take no TTFT sample on an early exit of a native Anthropic stream (PR #2072, epic #1624). A client-cancelled passthrough stream sampled continuum_backend_ttft_seconds, against the ledger rule that only a completed attempt reports time to first token.
• Decouple the W3C trace id from X-Request-ID and add tracing.headers.client_ref and x-continuum-request-id (#2104). The default middleware chain no longer stamps a hyphenated UUID onto inbound x-request-id when the client sent none, and TraceContext never adopts X-Request-ID or X-Correlation-ID as the trace id: trace_id is always 32 lowercase hex characters ([0-9a-f]{32}, not all zeros), taken from a valid inbound traceparent, else a 32-hex X-Trace-ID, else freshly generated, so the outbound traceparent sent to backends is always W3C-valid (^00-[0-9a-f]{32}-[0-9a-f]{16}-[0-9a-f]{2}$) and a valid inbound traceparent keeps its trace id and tracestate while minting a new outbound span id. Valid client X-Request-ID values ([A-Za-z0-9._:/-]{1,128}) are logged as x_request_id and echoed on the response x-request-id header without overwriting outbound X-Request-ID/X-Trace-ID/X-Correlation-ID, which carry the 32-hex trace_id. A new optional tracing.headers.client_ref field (default "", disabled; e.g. X-Client-Ref) accepts caller correlation references matching ^[A-Za-z0-9._:-]{1,128}$, forwards them to the backend, logs them as client_ref, and exports them on router.request as router.client_ref, while invalid values are stripped without failing the request and increment client_ref_rejected_total{reason} (empty, too_long, control_char, invalid_char). Every response inside the usage ledger scope (200 buffered and streaming, and 4xx/5xx errors) now carries x-continuum-request-id: req_<32 hex> matching the request's usage-record request_id and the router.request_id span attribute, and x-request-id echoes the client's valid X-Request-ID or falls back to x-continuum-request-id (earlier rejections outside the ledger scope omit x-continuum-request-id and echo only a valid client X-Request-ID). Migration note: log/trace pipelines that previously joined on a non-hex or UUID X-Request-ID inside trace_id or router.trace_id should read x_request_id in structured logs, x-request-id on responses, or configure tracing.headers.client_ref: "X-Client-Ref" for caller correlation references (client_ref / router.client_ref). See docs/en/trace-export.md and docs/en/configuration/advanced.md.
• Interrupted streams now charge what they generated (#2039, epic #1624 unit C2). Every early exit of the OpenAI-wire streaming relays (the chat transformer's OpenAI and thinking arms, the three Unix-socket relays, the SSE pipeline, and mid-stream fallback) records exactly one usage record with the output counted up to the exit: a client that cancels records client_cancelled (the engine's numbers when a usage chunk already arrived, else the router's delta count), a first_byte timeout timeout_first_byte with zero output, a total-budget or inactivity timeout timeout_mid_stream, a handshake error status or repeated upstream errors upstream_error with the status, and a guardrail block before the usage chunk guardrail_blocked, counting the deltas the gate held. Before this, an early return recorded nothing and a timeout or block was recorded as a normal end. A mid-stream fallback hop records the failed attempt's partial output as fallback_source (or upstream_error with zero output when it generated none) and the successor only its own generation, as attempt_seq 2 with parent_attempt 1. A direct (no-chain) stream now begins a ledger attempt, so its /admin/stats row names the backend, a handshake answered with a non-success status counts as one failed attempt with one record, and streaming_fallback_total gains a backend label naming the backend whose attempt failed. See "Stream exits and their usage outcome" in docs/en/architecture.md.
• Interrupted streams charge what they generated on the remaining relays (#2050, epic #1624 unit C3). Every early exit of the /anthropic/v1/messages stream arms (native passthrough over HTTP and a Unix socket, Bedrock runtime, and the OpenAI-wire and Responses-wire conversions), the /v1/responses pass-through and its Anthropic, Gemini, and Chat Completions conversions, the responses_only bridge, and the chat transformer's TCP Anthropic-source and Gemini arms now records exactly one usage record counted to the exit: a client that leaves is client_cancelled, a guardrail block guardrail_blocked, a read timeout timeout_first_byte or timeout_mid_stream by whether a frame arrived, and any other read error or a response.failed terminal upstream_error, with the engine's usage when it arrived before the exit and the router's delta count otherwise. On the Anthropic ingress a client that leaves while the upstream then stalls or fails is client_cancelled, not the timeout or the error, and gets no error event. A report that states the engine's input but not its output (an Anthropic message_start, or a Gemini usageMetadata without candidatesTokenCount, which Gemini sends only on its terminal chunk) is charged as that input beside the router's delta count, so a /v1/responses stream from an Anthropic or Gemini source cut before the output count no longer records zero output as engine_final. Before this, those exits recorded nothing, or a zero-token stats row, although the engine generated tokens up to the cut. Billing-visible changes: a guardrail block on a stream that carried no usage report now records the router's count instead of nothing (reversing the #1833 rule on the Responses conversions and the bridge); a Chat Completions conversion on /v1/responses that completes without engine usage records usage_missing; a response.incomplete for max_output_tokens records length; and an Anthropic ingress stream that ends after a data-level error event writes a stats row with success: false that still counts its tokens. A block the SSE pipeline's guardrail gate decides at [DONE] or at the end of the source (the Gemini and Bedrock arms of /v1/chat/completions) now records guardrail_blocked rather than a normal end. A failed attempt row (outcome other than completed, length, or usage_missing) reports no TTFT and takes no response_headers TTFT sample, and an /admin/stats request row written on a stream relay task now counts the failed attempts made before the stream. See "Stream exits and their usage outcome" in docs/en/architecture.md.
• Carry the router hop count on streaming output-guardrail checks and streamed action-safety decisions (#2054). Every streaming surface (/v1/chat/completions, /v1/responses, /anthropic/v1/messages, and the responses_only bridge) moves its output gate into a spawned relay task, and a tokio task-local is not inherited by a spawned task, so the chunked window checks, the buffer_full end-of-stream check, and the monitor-mode final check ran outside the request's hop-count scope and sent no x-continuum-router-hops header. The same held for the System One action-safety decision on streamed tool calls on /anthropic/v1/messages and the responses_only bridge. A guard target that is a router therefore restarted the count at zero, the fail_closed-on-508 rule of #2032 never fired on a streaming route, and a guard self-loop on a streaming route ended after MAX_ROUTER_HOPS + 2 inbound requests instead of MAX_ROUTER_HOPS + 1. StreamingOutputGate and ActionSafetyEvaluator now capture the count when they are built on the request task and re-enter it around every provider call they make from the relay.
• Keep a client-supplied reasoning_effort on the auto-selection streaming arm for a self-hosted OpenAI-compatible backend declared type: openai (#2030). #2015 scoped the legacy-model removal to the OpenAI cloud and Azure in normalize_openai_streaming_payload, but stream_with_auto_backend_selection builds its own payload and still called the unscoped normalize_openai_reasoning_request, so that arm removed the value for every type: openai backend: a local engine lost it there while the direct streaming route and the non-streaming route kept it. The arm now answers the same endpoint question as its sibling — Azure unconditionally, Openai only when the URL contains api.openai.com — so the three routes agree. The arm is library-only (it has no in-crate caller; it is pub through src/http/streaming/mod.rs and src/streaming.rs), so this reaches embedders. OpenAIBackend::transform_request, the other site #2015 left unscoped, is unchanged.
• Pin Dynamo shim sends from the Anthropic Messages and Responses ingresses (#2029). Both ingresses convert the request to a chat body and sent it to a --router-mode direct shim backend without the x-dynamo-* worker pins, so the frontend refused it. Every send to a shim-pool backend (buffered, streaming, each web_search loop round, retry, and fallback attempt, plus /v1/responses passthrough and responses_only sends) now carries the chat path's pin, a client nvext.token_data is always dropped on those sends, and a direct-mode refusal is answered 503 disaggregated_prefill_decode_unavailable in the ingress' own envelope. Not covered: a refusal inside an opened 200 SSE stream, a non-2xx on a Unix-socket streaming backend, and /v1/responses/compact.
• Pass a downstream router's structured 4xx through on /tokenize and /detokenize (#2024). A continuumrouter backend's refusal such as 400 tokenize_option_unsupported with param: continue_final_message reached the client as tokenize_upstream_rejected with a fixed message and no param. A downstream 4xx whose body is a router error object now keeps its status, code, type, and param, with the message capped at 256 bytes, so a chained deployment answers what a direct router does (a downstream 404 model_not_found now answers 404 model_not_found). A downstream 401, 403, or 407 still answers 502, any other body stays tokenize_upstream_rejected, and other engine types are unchanged.
• Apply logging.level, logging.format, and admin.audit.include_body, which the router validated and documented but never read (#2042). Every deployment logged at INFO in text whatever the file said, and CONTINUUM_LOG_LEVEL changed nothing; the logging keys now configure the process subscriber at startup (they stay restart-required), with compact as a third format. include_body: true now adds the JSON body of Admin POST, PUT, PATCH, and DELETE requests, up to 16 KiB and masked by key name with the configuration export masker (plus the new key value in POST /admin/api-keys and the configuration text in POST /admin/config/import and /admin/config/validate), to the admin_audit line; a body that is not recorded is noted as body_omitted (not_json, too_large with body_bytes_read, or invalid_json), a masked form over 16 KiB is cut and flagged body_truncated, and the handler still receives every byte. With include_body: false, the default, the line is unchanged.
• Complete the accounting seams for relayed streams (#2078, PR #2087). /v1/responses, the responses_only chat bridge, /anthropic/v1/messages, and the SSE pipeline now record each stream under its real terminal outcome in GET /admin/stats by_outcome (once, not also as completed), match concurrent streams to their connect-time rows by request_id, and feed GET /admin/usage/export and the aggregate token counters even when admin.stats.track_tokens is false.
• Enforce chunk_interval on the plain streaming relays and stop on a client disconnect during upstream silence (#2079, PR #2083). The chat transformer, thinking, and Unix-socket relays now end a stalled stream after the model's streaming chunk_interval as a mid-stream timeout, and the Anthropic, Gemini, and chat source relays drop the upstream as soon as the client leaves instead of waiting for the next byte. Usage on non-terminal /v1/responses events stays provisional until the stream ends.
• Validate the usage section at load and honor reconciliation in the Admin backend API (#2080, PR #2085). usage.quality_judgment and usage.tokenize_correction rules now apply at startup, hot reload, config sync, and Admin configuration writes, and continuum-router config validate reports them under their dotted paths. POST /admin/backends, POST /admin/backends/probe, and PUT /admin/backends/{name} accept reconciliation (default true on create), and a change to it counts as a backend change on reload.
• Preserve incomplete, refusal, reasoning, and failed outcomes on the /v1/responses chat bridge (#2122, PR #2124). A Responses reply that ended incomplete, failed, or cancelled, or that carried a refusal or reasoning item, was answered 502 responses_parse_failed or as an empty finish_reason: "stop" completion. Refusals now map to message.refusal and finish_reason: "content_filter", reasoning to reasoning_content, a content-filter truncation to content_filter, and a truncated function call to length, in buffered and streamed replies. A stream the upstream cut off before its terminal event now ends with an event: error frame (upstream_error) instead of a synthesized stop chunk and [DONE]. Chat Completions clients of responses_only models now receive reasoning summaries as reasoning_content, as Anthropic and Gemini backends already sent them; output guardrails inspect that text only with guardrails.inspect_reasoning: true.
• Discover Anthropic models live from GET /v1/models (#2089, PR #2114). Anthropic backends returned a fixed pair of Claude 3.5 ids, one of which does not exist. Discovery, candidate probes, and model aggregation now page through the key's catalog with x-api-key and anthropic-version, and fall back to a refreshed built-in catalog (with the error recorded on the probe result) only when the call fails. The built-in Haiku 3.5 id is corrected to claude-3-5-haiku-20241022.
• Accept map-encoded KV cache events from vLLM 0.29 and later in continuum-kv-listener (#2091, PR #2108). vLLM 0.29 encodes EventBatch and its events as tagged maps instead of arrays, and the listener rejected every batch; both encodings now decode to identical output. The listener also answers /health beside /healthz.
• Keep the local-only kv_routing fields across Hub config sync (#2090, PR #2111). Applying a Hub kv_routing section reset prefix_routing.salt_echo to false and kv_cache_index.backend to memory, so a router on kv_cache_index.backend: redis rebuilt its KV index in memory on every sync. Both fields now stay as configured locally.
• Attribute web-search tool-loop usage to the serving backend (#1968, PR #2016). A non-streaming chat request on a self-hosted backend with web_search enabled was recorded with backend unknown and no backend_id; the usage event, provider, and x-served-backend now name the backend that served the final turn.
• Harden the tokenizer routes and the count_tokens tokenize arm (#2001, PR #2017). The count_tokens engine call reads through the shared 16 MiB cap and falls back to the local estimate on an oversized body, tokenizer requests are admitted through a half-open circuit without taking a probe slot (a transport failure still reopens it), and token_strs is held as one buffer with offsets.
• Keep reasoning_effort on streaming requests to self-hosted type: openai backends (#2000, PR #2015). The streaming route dropped the field for any model outside the OpenAI reasoning families; the strip now applies only to api.openai.com and type: azure, which reject it.
• Keep an all-features debug build within the default 2 MiB thread stack (#2136, PR #2137). The chat ingress inlined its streaming, disaggregated, and semantic-cache futures into one 391 KB poll frame, so four all-features test targets aborted with a stack overflow. The largest awaitees are now boxed behind non-async wrappers, the task-local scope helpers return the scoped future, and ContinuumRouterBuilder::build returns a boxed future; .build().await is unchanged for callers.

CI¶

• Pin every third-party action reachable from a self-hosted runner by full commit SHA, and fail CI on a mutable ref there (#1488, PR #2037).
• Run a toolchain-free shell-syntax job when scripts or workflows change (#1489, PR #2035).
• Sync Dependabot pull requests to the project board by running the board workflow on pull_request_target (#2022, PR #2023).
• Bump astral-sh/setup-uv 10.1.0 to 10.3.0 (PRs #2012, #2075), actions/setup-go 6.5.0 to 7.0.0 (PR #2013), Azure/setup-kubectl 5.1.0 to 5.2.0 (PR #2076), and the dtolnay/rust-toolchain pin in the local toolchain action (PR #2077).

Documentation¶

• Re-capture the mlxcel tokenizer fixtures from mlxcel-server at commit 1e561f1e (#2020). From mlxcel#2127 onward, mlxcel answers an out-of-vocabulary id on /detokenize with 500 server_error, as llama.cpp does, so the router now answers such a request to an mlxcel backend with 502 tokenize_upstream_error instead of returning a 200 whose text silently lacks the id. The router itself is unchanged: an older mlxcel build still answers 200 and drops the id, and that answer still reaches the client as a 200. The mlxcel verification note and the 502 row in the tokenizer section of docs/en/api.md say so.

Dependencies¶

• Replace unmaintained unsafe-libyaml (RUSTSEC-2024-0449) by switching serde_yaml to serde_norway 0.9 across root and standalone crates (continuum-event-broker, continuum-kv-listener), and remove the advisory exception from deny.toml (#2084).
• Re-pin the vendored continuum-protocol crate to Hub revision 0165a2b8, which completed the usage batch hash chain, and clear its four LOCAL-AHEAD entries; the protocol bytes are unchanged (PR #2086).
• Bump the minor-and-patch group with 5 updates (PR #2014): lazy_static 1.5.0 to 1.5.1, redis 1.7.0 to 1.7.1, rmcp 3.4.1 to 3.5.0, tokio-rustls 0.26.5 to 0.26.6, and tokio-test 0.4.5 to 0.4.6.
• Bump the minor-and-patch group with 3 updates (PR #2038): tokio 1.53.1 to 1.53.2, uuid 1.26.1 to 1.27.0, and libc 0.2.189 to 0.2.190.
• Bump the minor-and-patch group with 6 updates (PR #2074): toml 1.1.6 to 1.1.7, rmcp 3.5.0 to 3.5.1, hyper 1.11.1 to 1.12.0, dynamo-kv-hashing 1.5.0 to 1.5.1, h2 0.4.19 to 0.4.20, and rust-embed 8.12.0 to 8.13.0.
• Update rustls 0.23.45 to 0.23.46 in the root and standalone crate lockfiles, which fixes RUSTSEC-2026-0346, RUSTSEC-2026-0347, RUSTSEC-2026-0348, and RUSTSEC-2026-0349, and refresh the other semver-compatible dependencies (PR #2137).
• Update etcd-client 0.20 to 0.21, minijinja and minijinja-contrib 2 to 3 (with the serde feature, which replaces Value::from_serialize), and tokenizers 0.22 to 0.23, and raise the hyper, http, and http-body requirements of the standalone crates (PR #2137).

v1.32.1 - 2026-10-05¶

Fixed¶

• Cancel a silent streaming /v1/chat/completions relay when the client disconnects (#2008). The OpenAI-wire, thinking-pattern, Anthropic, Gemini, Unix-socket, and responses_only chat relays awaited only the next upstream byte, so a client that left while the provider was silent (a long reasoning pause) kept the upstream request open until the provider emitted again or timeouts.request.streaming.total elapsed, and a single-slot local model kept generating for nobody. Each relay now watches the downstream close beside every upstream read and drops the upstream at once. With mid-stream fallback enabled, a client that leaves during a silent phase is reported as disconnected rather than as an inactivity failure, and no fallback backend is dialed for it. A connected slow reader is still charged from the terminal usage chunk (#1815).
• End a continuum-kv-listener push data stream cleanly when it falls behind its live feed before the router's response head arrives (#2010). The listener waited only for the head and then dropped the stream, so the router could see a reset instead of the request's clean end alongside the buffer_overflow close. It now also waits, within the same 2-second grace, for the router to close its side, as it already did once the head had arrived.

CI¶

• Run the tests from two builds, all features and default features, instead of rebuilding the library once per feature combination, in both make ci-local and the hosted ci.yml (#2005, PR #2007). A full local run dropped from 94 steps and 3616 s to 35 steps and 2387 s. make ci-local-gated (--gated) adds an opt-in local run that skips the steps of jobs the hosted gate would skip, and the test discovery audit now checks that the batched step covers every feature-gated target.

v1.32.0 - 2026-10-03¶

Added¶

• Add POST /tokenize and POST /detokenize (aliases /v1/tokenize, /v1/detokenize), which reach the tokenizer of the engine serving the requested model (#1998). vllm, sglang, llamacpp, and mlxcel backends are supported; the client contract is the vLLM and SGLang shape and the router translates it for llama.cpp and mlxcel (add_special_tokens defaults to true there, and messages is refused). Dispatch is exact-model and single-shot: no retry, no fallback chain, no cache, no model experiment, no usage record, and no per-model rate-limit charge. Both routes need the read scope; a model served only by other backend types answers 501 tokenization_not_supported. /anthropic/v1/messages/count_tokens now takes its engine paths and payloads from the same wire table. A backend URL ending in /v1 (for example http://host:8000/v1) has that one trailing /v1 dropped before the engine path is appended, so vllm, llamacpp, and mlxcel are called at /tokenize and /detokenize and sglang at /v1/tokenize and /v1/detokenize; before, such a URL produced /v1/tokenize or /v1/v1/tokenize for count_tokens, which the engines do not serve. Any engine 5xx on these routes answers 502 tokenize_upstream_error, an engine 401, 403, or 407 answers 502, and an engine 429 keeps its Retry-After. For the backend's circuit breaker, any HTTP response from the engine on these routes is neutral whatever its status, so a tokenizer 2xx cannot close or keep closed the circuit of a backend whose generation path is failing and a client-caused 5xx cannot open one; only a connection failure or a timeout counts, as a failure, and an open circuit still refuses the request. Engine answers are read from the capped 16 MiB body without a JSON tree, so the largest allocation for a dense id list is the 64 MiB id vector rather than a 256 MiB tree.
• Add backends[].responses_api: native so a self-hosted backend that implements the Responses API itself is served through passthrough without being declared type: openai (#1987). Any OpenAI-wire type accepts it; anthropic, bedrock, gemini, and a unix: URL refuse it at load. Nothing else the type decides changes, and the credential goes out in the backend's own scheme (Basic with auth.type: basic). On passthrough the router now forwards request fields it does not model (conversation, background, service_tier, and later additions) and input items of types it does not model (mcp_approval_response), where it used to drop the fields and answer 422 for the items; background: true is forwarded only on passthrough, where the queued POST records nothing and the first router-mediated GET or cancel of the terminal result records its usage once and, under a guardrail, gates it (a streamed background request is refused under a guardrail; a result never fetched through the router is not metered), and top_k, min_p, max_tokens, and max_completion_tokens sent there are refused while request_params or a hub tier limit governs them; a conversion strategy refuses conversation and such items with 400 unsupported_request_parameter, and a request carrying an unmodeled field is never served from the response cache. GET, DELETE, and the new POST /v1/responses/{id}/cancel on a response a passthrough backend served now reach that backend instead of answering 404: the router keeps a bounded per-router binding of response id to backend under the authenticated principal, re-checks the backend and the caller's allowed_backends against the live configuration on every call, and never sends an unrecorded id anywhere. The option reloads immediately; the Hub local export reports it as an unsupported field.
• Add the input_tokens smart-routing policy condition (#1988). A when.input_tokens range (gt, gte, lt, lte) matches the typical whole-request input-token estimate that fallback budget planning already uses, now shared from core::prompt_estimate: every message of the conversation rather than the latest turn, tools, functions, and response_format. A long prompt can reach a long-context tier on the first attempt instead of failing with 400 context_length_exceeded on a small-context primary. The estimate is computed once per request and only when a loaded policy reads it, is reported in the X-Smart-Route-Input-Tokens debug header, the Routing policy matched debug log, and POST /admin/smart-routing/simulate, and round-trips through GET/PUT /admin/smart-routing/policies. Validation refuses an empty range, two lower or two upper bounds, and a range admitting one estimate or none. It applies on /v1/chat/completions, the only ingress that runs smart routing; the hub policy contract has no member for it, so a size-gated rule is local.
• Add backends[].replay: enabled | disabled for backends whose requests have side effects, such as an agent runtime (#1993). With disabled, once a dispatch could have delivered the request, a dropped connection, a timeout, or a retryable status ends the request with that failure: no retry on the same or another backend, no fallback chain hop (the attempt reports the new replay_disabled trigger, which never hops), no pre-stream walk or mid-stream switch, no quality-check hop away from its reply, and no disaggregated unified fallback. A failed TCP or TLS connect is provably undelivered and still retries and falls back on the buffered routes; a streaming request selected onto the backend is dispatched once, a streaming chain hop onto it stops there unless the connect failed, and the Responses and Anthropic ingresses stop at any failure after committing to it. Requests such a backend may serve skip request deduplication, so independent requests sharing a trace id are no longer answered by one execution; the response cache never stores or serves their replies, Hub synthetic probes skip the backend, and an emulated /v1/decisions chain does not hop after it. Read per dispatch (immediate reload), accepted by the Admin API and the WebUI backend form, and reported as unsupported in the Hub local export. RetryState::backend_attempted and its two methods are deprecated: the router never recorded an attempted backend, so the selection filter that read the list never excluded anything, and it no longer reads it.
• Add smart_routing.unprofiled_models (#1975). default_tier (the default) keeps today's behavior; exclude admits only models a model_profiles entry matches (exact or pattern, local, hub-stated, or added through the Admin API), so a model that would join a tier through auto-inference or default_tier is never a target, and a policy tier with no profiled model takes the ordinary tier fallback. The option reloads immediately, config validate warns when exclude is set with model_profiles empty, and GET /admin/smart-routing/status lists the pool as candidate_models (with each model's tier and profile source) and the dropped models as excluded_unprofiled_models.
• Add an end-to-end test of the in-cluster event broker tunnel and a runbook for checking it on a live fleet (#1919, epic #1703). tests/dynamo_broker_tunnel_test.rs runs the router's event bus listener, the real continuum-event-broker binary (built from crates/continuum-event-broker into that crate's own target/ or $CARGO_TARGET_DIR/continuum-event-broker, or taken from CONTINUUM_EVENT_BROKER_BIN), and zeromq publishers of kv_metrics and kv-events, with a fake Kubernetes API server serving DynamoWorkerMetadata to the broker. It checks that the load store and the KV index end in exactly the state a direct dial of the same frames leaves, that a SIGKILLed broker leaves every subscription pending with stale but held load and no worker excluded until a new broker process resubscribes them, and that a publisher the metadata does not advertise is refused. scripts/dynamo-broker-live-check.sh and the new "Live check of the broker tunnel" section of the Dynamo integration guide describe the live check on kbm-g-np-kt-a (Pods reaching the bus, load freshness against dynamo.load.max_staleness, and relay latency); that check has not been run yet.

• Assign broker subscriptions by Dynamo namespace across several brokers (#1920, epic #1703). With dynamo.load.transport: broker, a kv_metrics or kv-events subscription now goes to one of the live broker sessions covering its publisher's namespace by a fixed rendezvous ranking (SHA-256 of the publisher target and topic with each broker id) instead of the least-loaded session, so subscriptions spread across equally eligible brokers, a reconnect returns to the same broker on every run and router replica, adding a broker moves no open stream, and losing one moves only its subscriptions, each to its next-ranked broker. A subscription whose broker session ends reconnects through a remaining covering broker after the floor of the backoff (at most 500ms, never escalated), so load freshness recovers within one reconnect; kv-events cursors carry across the move. A subscription waking from pending waits a random delay of at most 500ms before its first connect, and a kv-events subscription that stays without a connection for 60s has its ranks cleared from the KV index (logged). Under broker, GET /admin/dynamo/workers adds broker_coverage (per configured namespace: covered, the covering broker ids, and pending subscriptions per family), and two gauges report the same per configured namespace, bounded by the 16-namespace cap and removed when a namespace leaves the set: continuum_dynamo_broker_namespace_sessions{namespace} (0 = uncovered) and continuum_dynamo_broker_pending_subscriptions{family, namespace}. Router replicas still do not coordinate: a replica with no covering session keeps its subscriptions pending. The gauges are recomputed at most once per 250ms, so a mass failover does not make the reporter spin.

• Add dynamo.load.transport: direct | broker (#1917, epic #1703), shared by the kv_metrics load and kv-events KV subscriptions and defaulting to direct. With broker, each subscription is carried over a live dynamo.event_bus broker session whose granted namespaces cover the publisher's Dynamo namespace, and the router's own bounded ZMTP client, decoder, publisher-id check, sequence dedup, and kv-events gap handling run over the relayed bytes exactly as with a direct dial. A subscription with no covering session is pending: it dials nothing, its worker reads as stale load (round-robin, never excluded), and it starts as soon as a covering session is welcomed; when a broker's session ends, its subscriptions reconnect through the normal backoff and end pending if no covering broker remains. broker needs the dynamo-event-bus build feature; a binary without it warns at startup and keeps the subscriptions pending rather than dialling directly. The field reloads immediately and restarts only the subscriptions, never discovery. GET /admin/dynamo/workers gains subscriptions (family, publisher id, namespace, component, state, transport, and carrying broker id; never the publisher endpoint), and continuum_dynamo_broker_subscriptions{family, broker} counts the connected subscriptions each broker carries. A socket close after a receive error is now bounded at 1s so a stalled tunnel cannot hold a subscription task. Only broker-role sessions carry subscriptions, never kv_listener sessions, and only a change to the set of placeable broker sessions wakes pending subscriptions. A restarted kv-events subscription for the same publisher (a transport switch included) resumes its per-rank event cursors, so events missed while it was stopped or pending are handled as a gap, and the ranks of a publisher that will not resume are cleared from the KV index.

• Add the kv_listener push role on the event bus, so a continuum-kv-listener the router cannot reach dials out to the router and feeds the KV cache index (#1921, epic #1703). dynamo.event_bus.brokers[] gains role: broker | kv_listener (default broker, omitted when serialized so older files round-trip) and backends (kv_listener only: required, non-empty, each 1-64 bytes of [A-Za-z0-9_-], unique, at most 256; namespaces stays broker-only and each is a load error on the other role), and kv_cache_index.event_sources[] gains transport: sse | event_bus (default sse; endpoint becomes optional in YAML, required for sse and forbidden for event_bus, whose backend_name must be a valid source id used by no other source). The continuum-kv-listener gains an optional event_bus section (url, tls_server_name, ca_file, client_cert_file/client_key_file, client_id, credential_file/credential_env, reconnect_backoff, drain_deadline, connect_timeout) and keeps its SSE server unchanged; the push stream carries the same bytes as GET /events/{backend}, asserted against a shared golden fixture, and the router resumes from the last forwarded event id after a reconnect (a source with no live listener session is retried at reconnect_interval_ms and does not back off further, and one SSE record's data: is capped at 1 MiB like the line buffer). The wire contract stays at protocol version 1 with two additive changes: AnnouncedSourceTarget.last_event_id (the resume position) and a Hello::validate rule that refuses a kv_listener hello with no sources. Security: the event bus stays TLS-only with token-hash or mTLS authentication, a hello is accepted only for the role its credential's entry is configured for, a listener may announce and be commanded only for the backends its entry grants (a reload that narrows the grant or changes the role closes the session with forbidden), and the token is never written in YAML or logged. config validate and the startup log warn about a bus source with the bus disabled or ungranted and a grant no source consumes; bus sources are not exported to the control-plane hub config sync. See docs/en/architecture/kv-cache.md, docs/en/architecture/dynamo-integration.md, and config.kv-listener.yaml.example.

• Add deployment assets for the in-cluster event broker (#1918, epic #1703). deploy/helm/continuum-event-broker is a chart of its own, installed in the GPU cluster in the fleet's namespace, and deploy/kubernetes/event-broker/ is the equivalent standalone Kustomize bundle. Both run one Deployment of exactly one Pod per broker identity (two in the examples), because the router keeps one session per broker id and a second session with the same id supersedes the first. They render a Role granting only get, list, and watch on dynamoworkermetadatas.nvidia.com in each watched namespace, mount each broker's bearer token from an existing Secret (never rendered, never a value) and the router's CA bundle, probe /readyz and /healthz, keep one broker through a node drain, and apply a default-deny NetworkPolicy whose egress reaches only DNS, the API server, the router's bus address, and the workers' publishers. An emptied egress list removes its rule instead of rendering an allow-all one. The chart refuses a missing or plaintext bus URL, a broker without a credential Secret, and a repeated broker id. On the router side, dynamoEventBus in the router chart exposes the dynamo.event_bus listener (container port, a separate Service, the TLS Secret mounted outside the read-only config directory, and a NetworkPolicy ingress rule), and values-dynamo-event-bus.yaml is the complete overlay. scripts/validate-deployment-assets.sh lints and renders all of it and checks the RBAC, credential, probe, identity, and egress contract, the chart's refusals, and that the router's listener port, TLS mount, and configured paths agree. The deployment guide gains an "In-cluster event broker" section covering the router's listener, the broker credentials, and reachability from the cluster to the router.

• Add the router side of the in-cluster Dynamo event broker tunnel (#1915, epic #1703), behind the new dynamo-event-bus Cargo feature on top of dynamo (not in default/full, compiled into official release binaries) and the always-validated dynamo.event_bus section, which is off by default and independent of dynamo.enabled; a binary without the feature logs at startup that the broker listener is inactive. A router outside the Kubernetes cluster cannot dial the workers' Pod-IP ZMQ publishers, so an in-cluster continuum-event-broker (#1916) dials out to a dedicated listener at dynamo.event_bus.listen, a port of its own that may not collide with server.bind_address, serving the crates/continuum-event-bus wire contract (#1913): POST /bus/v1/session and POST /bus/v1/streams/{stream_id}. The listener speaks TLS 1.3 with ALPN h2 only and refuses plaintext on the first byte (an HTTP-looking connection gets a best-effort 403 tls_required). A broker authenticates with a bearer token whose SHA-256 is configured as brokers[].token_hash (the token itself never is), compared in constant time, or with an mTLS client certificate chaining to tls.client_ca whose DNS SAN equals its id; its hello must use role broker, a client_id equal to that id, and only Dynamo namespaces in the broker's namespaces allowlist (an empty allowlist accepts any namespace and draws a config validate and startup warning). One session per broker, at most max_sessions sessions (default 64, 429 too_many_sessions beyond it) and 4 x max_sessions connections (a connection with no request accepted within 10s of connecting is closed and counted as auth_timeout, and one is closed 30s after its last accepted request ended; refused requests never keep a connection open), at most max_streams_per_session streams (default 1024), data streams accepted only for stream ids the router commanded in that session, and per-stream buffers and HTTP/2 windows derived from max_session_buffer_bytes (default 64 MiB, 32 KiB per stream per direction) so a session never holds more than its budget in flight. Every field reloads without a restart: enabled starts or drains and stops the listener, listen restarts it, TLS files are re-read for new connections, and a removed broker, changed hash, changed client_ca, or narrowed allowlist closes only the affected sessions (credential_revoked or forbidden), a changed client_ca also stopping connections verified under the old CA from authenticating again; a listener that fails to start is retried every 30s. EventBus::open_stream is the command API the ZMTP connector (#1917) will use. New closed-label /metrics families: continuum_dynamo_event_bus_connections_refused_total{reason}, continuum_dynamo_event_bus_sessions, continuum_dynamo_event_bus_session_refusals_total{reason}, continuum_dynamo_event_bus_sessions_closed_total{reason}, continuum_dynamo_event_bus_streams, continuum_dynamo_event_bus_streams_opened_total, continuum_dynamo_event_bus_stream_refusals_total{reason}, continuum_dynamo_event_bus_streams_closed_total{reason}, and continuum_dynamo_event_bus_bytes_total{direction}. See docs/en/architecture/dynamo-integration.md and the dynamo-event-bus configuration template.

• Add continuum-event-broker, the in-cluster end of the event bus tunnel (#1916, epic #1703). A router outside the GPU cluster cannot reach worker Pod IPs, so the broker runs inside the cluster, dials out to the router's bus over HTTP/2 and TLS (https:// only, bearer credential from a file or the environment, never logged), and for each subscribe relays the Dynamo ZMQ publisher's raw bytes over one HTTP/2 stream without parsing ZMTP. It dials only tcp://host:port publishers advertised in the event_channels of DynamoWorkerMetadata in the Dynamo namespaces it covers, as seen by its own read-only list and watch (RBAC: get/list/watch on dynamoworkermetadatas.nvidia.com only), optionally limited to allowed_cidrs checked after name resolution, and never loopback, link-local, or other special-purpose addresses unless explicitly allowed; anything else is refused with target_not_advertised, target_not_allowed, or unsupported_target, logged, and counted. Each stream buffers at most the session's max_stream_buffer_bytes toward the router, all streams of a session together at most 64 MiB, each chunk counted with a fixed per-chunk overhead, and a stream that would exceed either bound is closed with buffer_overflow rather than stalling the publisher. The session redials with jittered exponential backoff, and SIGTERM sends draining and lets open streams wind down until drain_deadline. Opening a stream is bounded to fit under the router's 30 second stream-open timeout (upstream connects at most 10 seconds in total across resolved addresses, 5 seconds for a free HTTP/2 stream slot, 10 seconds for the router's response head), and on expiry the upstream socket is closed and the stream is reported as refused or closed. A rebalance drain redials after a small jittered minimum rather than at once, and repeated rebalances fall back to the normal backoff. The per-refusal warning log is limited to the first per error code per minute plus a suppressed count, while the refusal metric counts every one. Setting allowed_cidrs to the Pod CIDR is strongly recommended, because an empty list trusts every advertised address that is not special-purpose. Prometheus metrics (continuum_event_broker_*) with /healthz and /readyz are served on metrics_listen. Ships in the Linux release archives and the Debian package with the continuum-event-broker(1) man page, and as the ghcr.io/lablup/continuum-event-broker image. See config.event-broker.yaml.example and docs/en/architecture/event-broker.md.

• Add a per-backend http_version (auto, http1, http2) and stream-limit observability (#1925, epic #1703). Over HTTP/2 the router multiplexes every request to a host onto one connection, so a server's SETTINGS_MAX_CONCURRENT_STREAMS (100, 128, or 250 on common ingresses) caps concurrent requests and parks new ones inside the HTTP client while the engine sits idle. http_version: http1 gives that backend its own HTTP/1.1-only client and lifts the cap; auto (default) keeps the shared client, so existing configurations are unchanged, and http2 forces HTTP/2 prior knowledge. http2 on a unix: backend URL is a load error. The field is accepted by the admin backend API, listed in the config schema, and read per dispatch, so a hot reload switches new requests without dropping in-flight ones. New /metrics families: continuum_backend_http_in_flight{backend, http_version} (in-flight requests, sampled at scrape time) and continuum_backend_response_wait_seconds{backend, http_version, outcome} (time from handing a request to the HTTP client until its send settles, with outcome of response, error, or cancelled), so a stall shows as an in-flight plateau at the server's stream limit with a long wait tail. See docs/en/configuration/backends.md and docs/en/metrics.md.

• Report KV offloading effects per backend on /metrics (#1565, #1877). New continuum_backend_cached_prompt_tokens_total{backend} adds the cached prompt tokens the serving backend reported (prompt_tokens_details.cached_tokens, Responses input_tokens_details.cached_tokens, Anthropic cache_read_input_tokens), and new continuum_backend_ttft_seconds{backend, source} observes router-measured time to first token from provider dispatch, split by source into streamed_first_token and response_headers (a non-streaming call's header arrival), with buckets from 25 ms to 60 s that separate local prefix hits, KV connector hits, and cold prefills. Both record on the default release build, not only with the control-plane feature: the stream-end dispatch instant is now stamped in every build. Each request counts once; local cache replays, failures, and usage-less responses add nothing. Streamed /v1/responses now builds its stream-end usage context on every build rather than only while the Hub agent or a policy is active, so it records both families and, like streamed chat completions, adds its tokens to llm_tokens_total and model_experiment_tokens_total. The backend label uses the existing sanitizer and cardinality guard, and the /metrics scrape drops series of backends no longer in the pool. vLLM reports cached tokens only with --enable-prompt-tokens-details. The vLLM engine-stats adapter now also reads vllm:external_prefix_cache_queries and vllm:external_prefix_cache_hits into the new backend_engine_external_prefix_cache_hit_ratio gauge (KV connector hits such as LMCache, beside the local GPU backend_engine_prefix_cache_hit_ratio), absent when the engine runs without a connector; SGLang has no equivalent counter pair, so the gauge is vLLM-only. continuum_kv_index_routing_decisions_total gains a tier label (gpu_hot, storage_warm, none) naming where the chosen backend holds the prefix, returned by the same scoring call that picked it, so a concurrent request cannot drop it; queries that sum over the family are unaffected. The Grafana router overview adds panels for all four.

• KV-overlap-aware prefill selection for a Dynamo fleet (#1702, epic #1703), behind the new dynamo-kv Cargo feature on top of dynamo (in official release binaries, off in default/full, denied on the embed graph) and the new always-compiled dynamo.kv section, whose every field reloads immediately. With kv.enabled: true the router subscribes to each worker's kv-events channel (discovered from the same v1/event_channels/ records as kv_metrics, read over the same bounded ZMTP client under a 1 MiB frame ceiling for block batches, and decoded position-tolerantly like ActiveLoad) into a bounded per-fleet radix index keyed by tokens_hash from the root with per-rank ownership (1,048,576 blocks in all, 262,144 per rank, gated on the registry's live worker set and the data-parallel ranks each worker's card declares, with a rank's ownership of a block counted per engine hash so a block stored twice survives one removal, evicted when a worker leaves; an event_id gap clears the rank and counts gap, and batches under a dropped parent count orphan_parent until the chain re-roots, since there is no request-plane snapshot recovery), renders and tokenizes each request with the fleet's own tokenizer and chat template resolved from the model card's tokenizer / prompt_formatter / chat_template_file slots into a blake3-verified, content-addressed cache under kv.artifact_cache_dir (local paths always; kv.allow_http_artifacts for a worker's metadata URL, default on; kv.allow_hf_artifacts for the Hub, default off), and only when the file's checksum is in the operator's kv.pinned_artifacts (default empty: a chat template is fleet-authored code whose memory minijinja cannot bound, since fuel counts instructions and one concatenation can allocate without limit, so an unpinned file is never fetched, parsed, compiled, or rendered; the model scores without a KV term as artifact_unpinned, the log names the slot, URI kind, and the card's declared checksum once per model and artifact for review, a removed pin drops the loaded tokenizer built from it at once, and config validate warns when kv.enabled with nothing pinned and refuses a malformed entry by index), hashes the prompt with dynamo-kv-hashing 1.5.0 (XXH3-64 over little-endian u32 ids per full block, seeded by the request's cache_salt and LoRA name, block size from the card, at most two sizes of 16 to 65536 per request, hashed on the blocking pool beside the render) so its hashes equal the workers' tokens_hash byte for byte, and applies Dynamo's own cost formula with kv.overlap_score_credit, kv.prefill_load_scale, and kv.router_temperature (argmin with a cursor for ties at 0, upstream's softmax sampling above). A rank's backlog counts only from a sample fresher than dynamo.load.max_staleness: ranks with one outrank ranks without, and a fleet with none (a dynamo.vllm fleet publishes no active_prefill_tokens) is ordered by overlap alone, with ranks nothing tells apart rotating as round_robin_stale. Decode stays load-only unless kv.conditional_disaggregation is on, which implements the isl_bounding / prefill_load / isl_or_load bypass and the decode-busy gate and then sends only x-dynamo-worker-instance-id; a Dynamo 1.5.x direct-mode frontend refuses that while prefill workers are registered, which config validate warns about. kv.send_token_data sends the rendered ids as nvext.token_data (overwriting a client-supplied value); the prompt is rendered from the body as the shim pool is sent it, streaming global_prompts and the single-leading-system merge included, and the ids are withheld when the pool's replicas would be sent different bodies. A tokenizer not loaded yet, a prompt above the 1 MiB render bound, an empty index, or a card without capacity never excludes a worker: the load-only rule of #1699 serves that request. New closed-label families continuum_dynamo_kv_subscriptions{state}, continuum_dynamo_kv_events_total{result}, continuum_dynamo_kv_index_blocks, continuum_dynamo_kv_selection_total{result}, and continuum_dynamo_kv_artifacts_total{result}; continuum_dynamo_selection_total{reason} gains kv_aware and conditional_bypass; GET /admin/dynamo/workers reports kv_event_publishers. The dynamo-kv-aware template is embedded. The dependency decision is recorded in Cargo.toml: dynamo-kv-router was measured (16 new crates, no runtime) but not taken because its unbounded indexer panics on malformed fleet-authored input under panic = "abort", and dynamo-tokenizers / dynamo-renderer were not taken because they pull C and C++ backends; the pure-Rust tokenizers and minijinja they wrap are used directly. The kv-events fixture is synthesized from the pinned v1.5.0 sources with an independent encoder and asks for a real capture, because the only fleet available runs with use_kv_events=False.

• Discover a Dynamo fleet on Kubernetes from the DynamoWorkerMetadata resources its workers write and the EndpointSlices that say which are ready (#1701, epic #1703). dynamo.discovery.source: kubernetes shares the dynamo.discovery.kubernetes block with kubernetes_pods (namespace, dynamo_namespace, watch_reconnect_backoff; model is required by kubernetes_pods only and ignored here, with a config validate warning when set) and reads what Dynamo's own discovery daemon reads in Kubernetes discovery mode: each worker's spec.data carries the same endpoint, model-card, and event-channel records an etcd fleet registers, so the router hands them to the etcd reducer under the keys etcd would hold them at and a Kubernetes fleet yields the same workers, capacity, data-parallel ranks, retained cards, and kv_metrics load publishers as an etcd fleet; GET /admin/dynamo/workers reports "signals": {"load": true, "capacity": true}. A worker is published only while an EndpointSlice endpoint naming its Pod is ready, joined on targetRef.name as upstream's daemon joins it; a resource whose endpoint or card ids disagree with hash_pod_name of its name is refused whole and counted as continuum_dynamo_discovery_events_total{kind="mismatch"}, and a per-container resource of container discovery mode (owned by a Pod of another name) is refused and counted as kind="refused". Both resource types are listed and watched together, either watch ending relists both, and the reducer is fed by diff so an unchanged relist counts nothing and first_seen survives a readiness flip. The record's transport (a Pod IP in this mode) is held as under etcd and never exported. Both lists are read in pages of 500 that stop at 16384 objects, each watch has its own idle deadline and a watch that closes well before the server timeout is an error, as for kubernetes_pods, and nothing reaches the reducer until both watches are open. A model card over the 1 MiB retained-card ceiling is reduced to its typed subset when the resource is read, the held resources keep at most 256 MiB of record bytes in total (a new resource past it is refused and counted as malformed), a reducer key two resources share leaves the reducer only with its last contributor, an endpoint or card path too short to carry an id is a mismatch, and a resource whose spec is not an object is counted as malformed instead of failing the whole list. The Helm chart adds get/list/watch on dynamoworkermetadatas.nvidia.com and endpointslices.discovery.k8s.io to the same Role behind dynamoPodDiscovery.metadata: true, deploy/kubernetes/overlays/dynamo-metadata-discovery/ is the Kustomize opt-in, and a new dynamo-kubernetes template is embedded. Wire shapes are pinned to upstream v1.4.2 and checked against v1.5.0; the fixtures are synthesized from that source around the four live Pod names and ids, not captured, and ask for a real capture.

• Discover a Dynamo fleet on Kubernetes from its Pods alone (#1869, epic #1703). dynamo.discovery.source: kubernetes_pods (the default stays etcd) with a dynamo.discovery.kubernetes block (namespace, dynamo_namespace, model, watch_reconnect_backoff) lists and watches the Pods the Dynamo operator labels for Kubernetes discovery, needing only get/list/watch on pods, for clusters where the router may not read DynamoWorkerMetadata or EndpointSlices. A Pod is a worker while it is Running, Ready, and not terminating; its role comes from the operator's component-type labels, and its instance id is upstream's hash_pod_name (SipHash-1-3 with zero keys over the Pod name, masked to 53 bits), implemented locally and pinned to ids read from a live 1.4.2 fleet. Pods carry no model card and no load publisher, so this source provides no load and no capacity and the shim picker rotates round-robin; config validate warns, and GET /admin/dynamo/workers now reports source, signals, and each worker's decimal worker_id. Container discovery mode Pods are refused and counted as continuum_dynamo_discovery_events_total{kind="refused"}, and continuum_dynamo_discovery_connected gains a source label. A reload that switches the source or edits the active block replaces the watcher and empties the registry. The Helm chart renders a namespaced read-only Role and RoleBinding behind dynamoPodDiscovery.enabled (off by default), and deploy/kubernetes/overlays/dynamo-pod-discovery/ is the Kustomize opt-in. The dynamo section's field rules are now enforced on the load and hot-reload path, not only by config validate. The dynamo feature adds kube and k8s-openapi (rustls, no kube-runtime), both denied on the embed graph.

• Discover and pin across several Dynamo namespaces (#1910, follow-up of #1884 and #1870, epic #1703). dynamo.discovery.etcd.namespaces and dynamo.discovery.kubernetes.dynamo_namespaces list the DYN_NAMESPACE of every DynamoGraphDeployment one router fronts (unique, at most 16, exactly one of scalar or list per block; single-namespace files are unchanged), and dynamo.shim.pools: [{namespace, backends}] binds each namespace to its own --router-mode direct frontend replicas, required once more than one namespace is configured, one pool per namespace and no backend in two pools. Every discovered worker carries the namespace it was found under (the etcd key prefix, the metadata record path, or the Pod's nvidia.com/dynamo-namespace label, which the Pod source now selects with in (a,b)), its identity is (namespace, instance_id), GET /admin/dynamo/workers adds namespaces and a per-worker namespace, and continuum_dynamo_workers plus the three per-worker load gauges gain a namespace label. A request is dispatched only through the pool of the namespace its pinned workers belong to, a namespace whose pool has no usable replica is skipped rather than mis-pinned, a prefill/decode pair always comes from one namespace, and between namespaces serving one model name the picker's normal load comparison decides. kubernetes_pods models entries gain an optional dynamo_namespace, required with several namespaces. Hot-reloading a namespace in or out prunes only that namespace's workers and keeps the others published without a gap (transition namespaces); a source, connection, or Kubernetes namespace change still empties the registry. A reload never pins from a snapshot that lags the live configuration: the shim path offers the picker only namespaces in both, and pins nothing from a snapshot of another discovery source.

• Map worker components to models in the kubernetes_pods Dynamo discovery source (#1885, follow-up of #1869). One Dynamo namespace can host several worker pools serving different models, told apart only by the operator's nvidia.com/dynamo-component Pod label, and publishing every Pod under the single dynamo.discovery.kubernetes.model pinned requests for one model to workers of the other. The new dynamo.discovery.kubernetes.models list ({component, model} entries, mutually exclusive with model, exactly one of the two required under kubernetes_pods) publishes each ready worker Pod with the model its exact, case-sensitive nvidia.com/dynamo-component value maps to; a ready worker Pod whose component is not listed, or that has no such label, is held but not published and logged at debug once per appearance, without counting as malformed or refused. Entries need a valid label value and model name, a component may not repeat (two components may share a model), and the list holds at most 256 entries; errors name the entry, as in dynamo.discovery.kubernetes.models[1].component. A models edit replaces the watcher and empties the registry like a model edit. Like model, models is ignored by source: kubernetes (#1701), where config validate warns that it is set and editing it restarts nothing. dynamo.shim.model_map is not a substitute, since it only renames the model the picker looks for. Single-model files parse and behave as before.

• Attach a per-request quality judgment and the model-experiment arm to control-plane usage records (#1728, epic #1719; protocol change continuum-hub#1264). The optional usage.quality_judgment block (a new top-level usage section) asks a System One decision model, after a non-streaming /v1/chat/completions reply passes every guardrail stage, whether it answers the request, refuses it, is written in the request's language (asked only when the request has letters), and, only when include_tool_results: true and the current turn carried tool results, whether it agrees with them. The answers ride on that request's usage record as quality (answered_bps, refusal_bps, optional grounded_bps and language_match_bps, integer basis points from round(p * 10000), plus the resolved judge_model such as jev-1.13.0); a missing or non-finite answer, or one failing QualityJudgment::validation_problem, drops the judgment and keeps the record. The client response never waits, and the record's accounting (router_usage_seq, cost-center and model-budget charging) and its push never wait either: the judgment fills a slot the usage loop seals when it forms the record into a push batch, so a judgment that finishes later is dropped and counted late rather than re-sent under the record's idempotency_key. sample_rate (default 1.0), models, head-and-tail state caps (512 / 1,536 / 1,024 bytes), and a per-router max_in_flight cap (default 32, excess shed and counted, never queued) bound the cost, and only the last user message and the reply (plus tool results on the opt-in) reach the decision model. Every usage record of a model-experiment request now carries experiment_id and experiment_variant_id, with or without a judgment. New metric usage_quality_judgments_total{outcome}, new template usage-quality-judgment. The vendored continuum-protocol crate mirrors the hub's additive QualityJudgment and the three UsageRecord members byte for byte; PROTOCOL_VERSION stays 0. Streamed replies (judged since #1801), the /anthropic/v1/messages, /v1/responses, and /v1/completions ingresses, and non-streaming chat served through disaggregated_serving or the Dynamo shim are not judged. Late judgments are the slowest ones, so a high late share skews the attached judgments toward fast requests; a longer control_plane.usage_push_interval_secs or a faster self-hosted target reduces it.

• Judge streamed /v1/chat/completions replies for usage.quality_judgment (#1801, follow-up of #1728). A streamed request is sampled right after the input guardrail and judged the moment its stream-end usage record is emitted, over the head and tail of the text the client read, kept in memory bounded by max_reply_bytes and capped exactly as a buffered reply is (reasoning text is never read). Because the record and the whole reply exist at the same moment, a target that answers well inside control_plane.usage_push_interval_secs leaves almost no late judgments. A streamed reply is not judged, and counts skipped, when the output guardrail cut or rewrote the stream (buffer_full or chunked block, or masking), a mid-stream fallback hop spliced two generations into one reply, the stream carried an error or timed out, or the client disconnected. usage_quality_judgments_total gains a stream label ("true" or "false") so the two modes share one ratio. Streamed chat served by a native Anthropic, Gemini, or Bedrock backend, over a unix socket, or through disaggregated_serving or the Dynamo shim is still not judged, nor are /anthropic/v1/messages and /v1/responses (#1803, #1804).

• Judge /anthropic/v1/messages replies for usage.quality_judgment, buffered and streamed, on every backend path: native Anthropic over HTTP or a unix socket, Bedrock mantle, runtime, and converse, the OpenAI-wire conversion (Gemini included), and responses_only models (#1803, follow-up of #1728). Tool results travel inside user messages on this wire, so the judged user turn is the last role: user message that carries no tool_result and has non-blank text other than Claude Code's <system-reminder> blocks, the rule the action-safety evaluator uses; reminder blocks, the top-level system, role: system messages, and thinking blocks never leave the router, and tool_result text leaves only under include_tool_results. The reply is its text blocks as the client receives them: a tool_use or server_tool_use reply, a pause_turn stop, an output-guardrail block, a masked or cut stream, a stream that relays an error event, and a client disconnect (including one after the last event) count skipped, while a max_tokens stop is judged. The web_search_20250305 server-tool emulation and response-cache hits are not judged.

• Judge /v1/responses replies for usage.quality_judgment, buffered and streamed, on every strategy: passthrough (to openai and azure backends, including responses_only models, and to any backend with responses_api: native since #1987) and the chat, Anthropic, and Gemini conversions (#1804, follow-up of #1728). The judged user turn is the last user message of the request's own input with non-blank text other than harness text (<system-reminder> and Codex harness messages such as <user_instructions>, <environment_context>, and # AGENTS.md instructions), so a previous_response_id continuation is judged on its delta only, and one whose input holds only function_call_output items counts skipped; instructions, system and developer messages, and reasoning items never leave the router, and tool output leaves only under include_tool_results. The reply is the output_text of its message items as the client receives it (a buffered reply the output guardrail masked is judged on the masked text), read from the terminal response.completed or response.incomplete event when streamed: a client-executed call item, a failed or cancelled status, an output-guardrail block, a masked or cut stream, and a client disconnect count skipped, while a status: incomplete reply is judged. /v1/chat/completions requests bridged to an upstream /v1/responses for a responses_only model are now judged too, buffered and streamed (the streamed half rides the stream-end record #1809 added). Response-cache hits and /v1/responses/compact are not judged.

• Add a client-facing System One decisions endpoint, POST /v1/decisions with /v1/systemone as an alias (#1726, epic #1719). A model named under the new decisions.targets[] (a name plus the shared systemone block) is passed through to that System One server and its answer returned unchanged except that usage.output_tokens is reported as 0; any other name is emulated on the chat model of that name as one forced emit_decision tool call at temperature: 0, answered with one-hot probabilities, no confidence, and emulated: true. The question map is validated before any dispatch (a 256-option choice answers 422, a padded model answers 400), the route needs the write scope in blocking mode, per-key and per-model rate limits, allow-lists, and budgets apply as on chat completions, a key with a non-empty allowed_backends never reaches an inline-endpoint target, an emulation hop runs chat's request_params policy and input guardrails and resolves aliases and model experiments like chat, usage is recorded on the caller's key under the model that served, and a target name works in fallback.fallback_chains, so a failing real target can fall back to an emulated chat model. The section is optional and reloads immediately; absent means enabled with emulation only. New decisions-endpoint configuration template.

• Govern smart routing policy and guardrail provider content from Continuum Hub (epic #1706, hub epic continuum-hub#1212). v1.32.0, this release, is the first router version that executes both: smart_routing under the new smart_routing_policy_v1 capability, and provider_content inside the existing guardrail_policy_v1 statement and its guardrail_policy_digest, advertised under the new guardrail_provider_content_v1 capability (#1818). A router older than v1.32.0 advertises neither smart_routing_policy_v1 nor guardrail_provider_content_v1, so the hub reports both as unsupported. A hub gated on guardrail_provider_content_v1 never sends provider content to a v1.17.0 to v1.31.x router, which keeps receiving the rest of the guardrail policy (mode, stages, thresholds, and routes); only against a hub that sends provider content ungated does such a router refuse the whole guardrail statement as digest_mismatch and keep its previous guardrail policy. A hub that predates hub#1212 sees no change beyond the new capability strings. No statement in either stream can reach provider or backend endpoints, credentials, api_key_env, bypass_api_keys, streaming buffer tuning, or the classifier backend's endpoint or credential, weaken a local enforce, or override a local smart_routing.enabled: false. The new Smart Routing Governance page, the provider content sections of the guardrails guide, and the governance compatibility matrix in the architecture guide describe the surface, and three new integration suites pin it: control_plane_governance_sovereignty_test, control_plane_governance_compatibility_test (restart and mixed-version behavior), and control_plane_governance_docs_examples_test, which runs the documented worked examples (#1713).

• Advertise guardrail_provider_content_v1 to Continuum Hub (#1818, hub epic continuum-hub#1212). A router whose guardrail policy reconciler is attached lists it in RouterPolicyStatus.capabilities on enrollment and every heartbeat, directly after guardrail_policy_v1 and on the same latch, so the two appear together or not at all. The hub gates provider_content on it (hub#1221): a router that does not advertise it receives the guardrail statement without provider_content, digested over that stripped policy, and the hub reports the content as unsupported for that router. The statement shape, its digest, the refusal codes of #1709 and #1711, and PROTOCOL_VERSION (0) are unchanged, and a statement without provider_content is still accepted and acknowledged. The vendored continuum-protocol crate adds GUARDRAIL_PROVIDER_CONTENT_CAPABILITY locally ahead of an upstream re-sync (hub#1215).

• Advertise smart_routing_policy_v1 to Continuum Hub (#1712). A router whose smart routing is governed by the hub reconciler (a control-plane build started from a configuration file with control_plane.policy enabled) now lists the capability in RouterPolicyStatus.capabilities on enrollment and every heartbeat, so the hub stops reporting it as unsupported and derives pending, active, or error from the acknowledged smart_routing_policy_digest and the typed rejection. The capability is advertised on the same latch that turns the acknowledgment on, so it never appears without the execution-gated acknowledgment and typed rejection of #1711. A router whose local smart_routing section has enabled: false still advertises it and reports every hub policy as smart_routing_unavailable (an authoritative clear is still acknowledged), matching how guardrail_policy_v1 stays advertised on a router whose guardrails cannot run a policy. guardrail_policy_v1 and its digest acknowledgment are unchanged and already cover provider_content, which rides the same digest; the separate guardrail_provider_content_v1 capability (#1818) is what tells the hub that a router executes it. A hub that never sends smart_routing sees no other change.

• Report refused hub smart routing and guardrail statements with typed codes, and acknowledge a digest only after it runs (#1711). A smart routing body refused when the envelope is applied is reported on RejectedPolicyRevision.smart_routing_policy_digest as invalid_smart_routing_policy (failed validation, or a tier or classifier method this version cannot interpret) or digest_mismatch, and never reaches composition; a valid statement this router cannot execute, including one composed onto a local section with enabled: false, is reported as smart_routing_unavailable in place of an active digest. For both smart routing and guardrails, an accepted digest is reported active only once the reconciler has recorded executing it (a clear, once the hub layer is dropped), so a heartbeat sent between an envelope apply and the next reconcile pass reports the statement as pending instead of claiming it. The single rejection slot carries a guardrail or smart routing digest only under that track's own code. After an execution-time refusal both tracks run local configuration alone, not an earlier hub policy held in memory.

• Report read-only smart routing and guardrail provider inventory to Continuum Hub (#1710). Enrollment and every heartbeat now carry RouterInventory.smart_routing_inventory (enabled, the policy and explicit profile counts, the classifier method, the LLM classifier backend name when the method can call it, virtual_model, and intercept_all) and RouterInventory.guardrail_provider_inventory (each configured provider's name, type, stages, order, and whether it runs). Both are read from the live pipeline, so a reload, an Admin API edit, or an applied hub policy shows on the next beat, and neither is gated on a capability. An absent member means unknown, [] means no providers, and enabled: false means configured but not running; a list that does not fit the bounds is left out whole with one warning rather than truncated. No prompt, pattern, routing condition, endpoint, or credential name is sent. This is an additive local protocol change ahead of an upstream re-sync (hub#1213); PROTOCOL_VERSION stays 0.

• Compose hub-authored guardrail provider content into local guardrails (#1709). A guardrail_policy_v1 statement's provider_content is keyed by the name of a provider this router already runs: a policy_prompt is appended after a custom_classifier row's local prompt under a fixed router-owned joiner, pii default_action and actions take the stricter action per entity (block over mask over allow) and a stated placeholder_format wins, deny_patterns under the reserved key match_list_deny block next to every local deny rule, and allow_patterns under match_list_allow only narrow the local allow lists. Governed patterns are compiled once per statement into one bounded RegexSet per list (256 KiB per pattern, 8 MiB per statement, a 2 MiB lazy-DFA cache), are never written into configuration or shown by GET /admin/guardrails, and disappear with a hub clear. Content naming a provider this router does not run, a member with no local target, or a target that never runs is refused whole as guardrails_unavailable, and patterns outside the compile limits as invalid_guardrail_policy; the router then runs its local configuration alone. match_list_deny and match_list_allow are now reserved provider names: a local provider row using either fails to load with reserved_provider_name.

• Apply a hub-distributed smart routing policy (#1708). On a control-plane build with control_plane.policy enabled, the smart_routing member of the policy envelope is now verified, stored, and composed with the local smart_routing: section: the hub's routing rules and model profiles apply when it states them, a local enabled: false always wins, intercept_all is on only when both sides say so, and a hub classifier backend is only a name resolved under the existing smart_routing.classifier.llm.backend rule. A statement the router cannot execute is declined whole and the local section keeps running. While hub policy governs smart routing, local reloads and PUT /admin/smart-routing/policies apply through the composition.

• Add the smart routing policy and guardrail provider content wire surfaces to the vendored protocol crate (#1707), locally ahead of an upstream re-sync (hub#1214, hub#1215). PolicyEnvelope.smart_routing carries a SmartRoutingPolicy (routing rules, model tier profiles, and a classifier selection whose backend is only a name) with the same tri-state as guardrails, its own smart_routing_policy_digest, the smart_routing_policy_v1 capability string, smart_routing_policy_digest on AppliedPolicyRevision and RejectedPolicyRevision, and the invalid_smart_routing_policy and smart_routing_unavailable codes. GuardrailPolicy.provider_content carries policy prompts, pii actions and placeholder, and deny/allow patterns keyed by provider name, covered by the existing guardrail_policy_digest. Each member is bounded at 256 KiB of canonical encoding with a matching term in the policy-sync response ceiling, a policy_prompt that looks like a URL or a credential refuses the whole statement, and an envelope that uses neither member is byte-identical to one from an older hub. PROTOCOL_VERSION stays 0, and the router-authored wire fixtures are to be replaced with the hub's copies on the next re-sync.

• Make an absent usage-record ttft_ms diagnosable (#1763). Every control-plane usage record now carries exactly one of ttft_ms and the additive ttft_absent_reason protocol field (TtftAbsentReason: batch, no_first_frame, transport_unobservable, not_observed), so the hub can tell absence by design from a timing the router failed to observe. into_usage_record enforces the invariant: a reason on a record that carries ttft_ms is dropped, and a record with neither gets not_observed. The batch completion tap stamps batch, a dispatched stream that ended without a provider frame stamps no_first_frame, and the non-streaming tap stamps transport_unobservable or not_observed from the upstream-TTFT cell. Two non-streaming seams that used to report no TTFT now report a real response_headers span: the Unix-socket transport, whose buffered client stamps the instant the response header block arrives (UnixSocketResponse::headers_received_at), and Bedrock Runtime / Converse, whose SigV4 client marks around its own non-streaming send. PROTOCOL_VERSION stays 0; this is an additive local protocol change ahead of an upstream re-sync, and hub-side consumption is continuum-hub#1164.

• Hop to the next fallback model when a non-streaming chat reply is well-formed but unusable (#1727). The optional fallback.quality_check block asks a System One decision model (hosted Jev or a self-hosted laya-serve) whether the reply answers the request, refuses it, or is cut off, over only the last user message and the reply (each head-and-tail capped, 512 and 1,536 bytes by default to fit Laya's multilingual context; system prompts and tool results never leave the router), and when 1 - answered, refusal, or truncated reaches threshold (default 0.8) it advances the existing chain. It ships in mode: shadow, which judges and records without hopping, because the threshold is not yet measured; set mode: enforce after reading the would-be trigger rate. The hop counts against max_fallback_attempts, runs with its own fallback_timeout_multiplier window, never exceeds max_extra_hops (default 1), re-applies the key's allowed_models, and must fit the non-streaming chain budget (the primary's standard.total times streaming_fallback_budget_multiplier). A reply cut at the client's own max_tokens never triggers on truncation. Guardrail blocks short-circuit before the check, a hop's reply passes the same output gate, and every failure serves the latest reply that passed it. A dispatched hop sets x-continuum-quality-fallback: 1 and counts fallback_quality_triggered_total{reason}; a triggered request records two usage rows, one per generation. The question battery and state builder live in services::answer_quality for reuse. New template fallback-quality-check.

• Fail CI when the vendored continuum-protocol crate drifts from its hub revision (#1795, Phase 3 of epic #1796). crates/continuum-protocol/UPSTREAM.sha256 records the hub's per-file SHA-256 digests under src/ and tests/ at the revision on the crate's Cargo.toml provenance line (and the digest of the hub's own crate Cargo.toml), and scripts/check-vendored-protocol.sh fails, naming each path, on a revision mismatch, a modified, missing, or added file, or a malformed or stale crates/continuum-protocol/LOCAL-AHEAD entry (the allow-list of files carried ahead of the hub, each naming the hub issue that upstreams it). It also refuses what could feed the build from outside that comparison: another crate-root entry, a crate Cargo.toml outside an allowlist of shapes (only [package], [features], and registry dependency tables, no quoted or dotted keys, no target, build-script, path, or git keys), a target cargo metadata reports outside src/ and tests/, and a router dependency other than path = "crates/continuum-protocol". --base <commit> fails a manifest edit that kept its # rev: line, a vendored file edited together with its digest; CI passes HEAD^1 and make ci-local the merge base with origin/main. With CONTINUUM_HUB_DIR set it also regenerates the manifest from a hub clone. The check needs no network or hub credentials and runs first in the ci-release-graph job and make ci-local, next to scripts/test-check-vendored-protocol.sh, which proves each failure mode on throwaway trees; crates/continuum-protocol/** now triggers the Rust CI jobs, so a fixture-only change is checked too. scripts/refresh-vendored-protocol-manifest.sh --hub <clone> --rev <sha> [--sync] regenerates the manifest and re-vendors the crate from a local hub clone, refusing to --sync over uncommitted work unless given --force.

• Use Hub-delivered per-user web-search credentials (#1636, PR #1966). A control-plane build with a web_search section advertises search_credentials_v1 and applies the policy envelope's search_credentials member: a request made with a Hub key searches with its owner's provider and credential (serper, brave, or exa) in preference to web_search.api_key, a tier with web_search_enabled: false gets no search tool, and once the Hub has delivered a set, an owner without a credential on a deployment that cannot authenticate gets none either. The credentials are held in memory only, indexed by the key's user, and never written to the key-table sidecar; a statement with more than 1,000 entries, a duplicate user, an unknown provider, or a missing or mismatched digest is refused whole. Each search is reported on the usage record as web_search_queries, web_search_provider, and web_search_credential_source. Requests without a Hub key, and Hubs that deliver nothing, behave as before.
• Persist the synced key table for Hub outages (#1453, PR #1965). With the opt-in control_plane.policy.persist_key_table: true (bounded by key_table_max_age_secs, default 86400, 60 to 604800), the router writes an allowlisted projection of the applied key table (key ids and hashes, tier bindings and limits, owners, revocations, rotation overlaps, usage snapshots, cursor, enrollment identity) to the owner-only <state_file>.policy-keys.json and restores it at agent startup, so a router restarted while the Hub is unreachable keeps admitting Hub keys instead of rejecting every one. Provider, backend-scoped, and search credentials are never part of the file. A sidecar from another enrollment, past its bound, wider than 0600, or malformed is ignored; a restored table is never reported to the Hub as applied and never outranks a Hub delivery. GET /admin/control-plane/status adds policy_sync.source (synced, restored, none), policy_sync.restored, and policy_sync.key_table_persistence.
• Sync model fallback configuration from Continuum Hub (#1632, PR #1923). control_plane.config_sync gains a closed schema-1 fallback section beside request_params and kv_routing, with durable apply, rollback, restoration at restart before listeners open, value-free observations, and validation of the effective chain graph. Local fallback policy, chain records, and per-model settings stay atomic pins, remote content is bounded (64 model keys, 10 targets per chain, 64 trigger codes), and fallback.quality_check stays local. Each admitted request keeps the configuration snapshot it entered with. Companion Hub change: continuum-hub#1314.
• Ship continuum-kv-listener in the router container images (#1626, PR #1912). Dockerfile.ci copies the router and the listener from the same verified musl bundle and keeps the 65532:65532 image user, so a deployment no longer maintains a derived image for the listener; the deployment guides describe architecture selection, the entrypoint override, and listener health probes.
• Add GET /health/ready, an unauthenticated readiness endpoint that answers 503 until at least one enabled, non-internal backend is active (#1558, PR #1808). The Helm and Kubernetes examples use it as the readiness probe while /health stays the liveness probe; like /health it is exempt from rate limiting. A control-plane replica that boots with an empty local backend section stays unready until its first Hub assignment reaches the pool.
• Add make hub-compat, which builds this checkout with the control-plane features and runs Continuum Hub's own scripts/router-compat.sh against the binary in its floor and latest modes (PR #1858, epic #1796). It needs CONTINUUM_HUB_DIR pointing at a hub clone that contains the vendored protocol revision, and no GitHub credentials.

Changed¶

• continuum_dynamo_load_subscriptions and continuum_dynamo_kv_subscriptions gain a transport label (direct, broker) and a pending state (#1917). A query that selects {state="connected"} now returns one series per transport; wrap it in sum without (transport) to keep a single series.

• .github/workflows/dynamo-e2e.yml builds the router image it tests from the dispatched commit by default (#1776, epic #1703). The newest release predated the router fixes the harness exercises and ghcr.io/lablup/continuum-router:latest refused anonymous pulls, so the workflow could only reach the pull failure. A new build_router input (default 1) compiles a static musl binary with the release feature set inside rust:<pinned channel>-alpine, packages it with Dockerfile.ci as ghcr.io/lablup/continuum-router:e2e-local, and prints the commit it was built from; build_router=0 pulls the .env pin as before. The runner_labels default moves to a GitHub-hosted ubuntu-latest runner, which pulls the pinned Dynamo image (about 0.7 GB compressed) in minutes, and the job timeout rises to 90 minutes to cover the cold release compile. scripts/e2e-dynamo-shim.sh gains SKIP_ROUTER_PULL=1, which pulls every image but a locally built router's, since SKIP_PULL=1 also skipped the Dynamo and etcd pulls.

• Run every feature-gated integration test target in CI and in make ci-local (#1898, supersedes #1792). A file gated with #![cfg(feature = ...)], or a gated test or inline mod, compiles to nothing under the default cargo test --tests step, and nineteen targets, the control_plane group's security audit among them, had no cargo test --features <F> --test <target> step in .github/workflows/ci.yml or scripts/local-ci.sh. Each now has one in the job that already builds its feature graph; routing runs twice, under control-plane and, filtered to its Redis KV index cases, in the live redis-shared-state job. A second audit in tests/cargo_test_discovery_audit_test.rs fails the default suite when a gated target lacks a step in either file, and local-ci.sh lists its control-plane steps literally instead of in a loop so both files share one shape. The CI graph gate now schedules a test file's #[path] loader in another target and derives the class of any tests/<class>/ or tests/<class>_*/ group directory. Also fixes a stale assertion in bedrock_converse_integration_test that the missing step had hidden, the dead-port race that made unhealthy_backend_is_never_selected flaky (the port is now held by a bound, non-listening socket), the rate-limit metrics tests that asserted nothing, and 28 test path references #1620 left stale.

• Group most integration tests into domain test targets so cargo test --tests links far fewer binaries (#1620). Every top-level tests/*.rs file links the whole library, so the link step dominated the suite. 194 of the 285 top-level files now live as unchanged modules under 18 group targets (tests/<group>/main.rs: server, auth, admin, models, anthropic, appproxy, routing, providers, tracing_capture, cache, responses, control_plane, fallback, systemone, files_media, guardrails, streaming, web_search), and tests/tool_calling/ became its own target, so the suite builds 109 integration binaries instead of 285. Files that CI or a script selects by name, that carry a runnable --test selector, or that read process-global state (the metrics registry, control-plane trackers, the config history store, the prompt file cache, the candidate-probe limit, the timeout boot pin) stay top-level. A grouped test is selected with cargo test --test <group> <old_stem>::, a substring filter that also selects any sibling whose name ends with the same stem (cargo test --test <group> -- --exact <old_stem>::<test> selects exactly one test); CLAUDE.md lists the groups and how to choose one. The CI feature-graph gate still maps a moved file to its graph by name, and treats a group's main.rs change as a change to its members.

• Gate the hosted CI feature-graph jobs on the files a change touches (#1621). A new Detect Changes step, scripts/ci_graph_gates.py, decides ci-isolated-graphs, ci-release-graph, and redis-shared-state separately from path classes, every feature = "..." string in the base and the head version of each changed file, the gates of the feature-gated modules that contain it, the files that include a changed data file or read it at runtime from a feature-gated test, and the --test targets each job names, so a pull request confined to ungated code such as src/services/smart_routing/ runs only ci-default. The step widens to every job whenever it cannot decide (a deleted or renamed file, a shared test helper, an unclassified feature, a tracked symlink under a code root, a diff it cannot compute, or a file the paths filter reported that git did not), a failed, skipped, or timed-out (5 minutes) gate step counts as "run", and workflow_dispatch still runs the full matrix. ci-default now always runs the gate's tests (scripts/test_ci_graph_gates.py), cargo check --features "control-plane appproxy-router hot-reload redis-cache otel dynamo" --lib, and the bare-graph clippy pair moved from ci-isolated-graphs, and when the gate skips the release graph on a pull request it runs that job's all-features clippy minus the integration test targets the change did not touch, so a core change still cannot stop bedrock-sigv4, s3-cache, or appproxy-legacy code, a feature-gated unit test, an example, or a bench from compiling. The gate runs on pull requests only; redis-shared-state stays ungated on a push. The rust filter now also covers non-Rust compile inputs (src/**, tests/**, examples/**, benches/**, docs/shared/llm-skills/**, config.yaml.example) and the CI scripts. make ci-local applies no gating and still runs every step, with the gate test, the union check, and the bare-graph pair now in its default section.

• Run the hosted CI feature-graph jobs only on pull requests and manual dispatches, and cancel superseded pull request runs (#1619). .github/workflows/ci.yml gains a workflow-level concurrency group keyed by event and by pull request number or ref: a newer push to a pull request cancels that pull request's in-progress run, while push and workflow_dispatch runs never cancel a running one and a newer queued run replaces an older pending one. A push to main or develop now runs ci-default and the ubuntu-latest jobs only, with ci-isolated-graphs and ci-release-graph skipped, because the pull request run builds the feature graphs for the PR's merge ref and the post-merge push no longer rebuilds them; a break that only appears after merge (a pull request merged before its run finished, or two individually green pull requests interacting after a squash onto a moved main) surfaces on the next pull request run that builds the graphs, on a workflow_dispatch, or in make ci-local. A new workflow_dispatch trigger runs the full matrix on demand, with Detect Changes reporting every path group as changed. make ci-local still mirrors the full pull request matrix, and dependency-float.yml is unchanged.

• Replace the synthesized kv_metrics fixture with two messages captured from a live Dynamo 1.4.2 dynamo.vllm worker and record a live-fleet compatibility table (#1749, epic #1703). src/dynamo/load/fixtures/kv_metrics_frame_captured.bin (sequence 5, kv_used_blocks 4) and kv_metrics_frame_captured_idle.bin (sequence 6, kv_used_blocks 0) are one worker's busy and idle messages, read-only through its own ZMQ PUB socket, and their provenance note records the fleet, versions, trigger, and decoded values. The decoder reads exactly what the worker sent, and the test encoder reproduces both messages byte for byte, so the real envelope's key order and integer widths match the encoder's. The capture shows a divergence the synthesized fixture hid: a real 1.4.2 worker sends both active_decode_blocks and active_prefill_tokens as nil and reports load only through kv_used_blocks; the decode cost's existing kv_used_blocks stand-in keeps decode selection load-aware on such a fleet (now pinned by a picker test fed the capture), while under the load-only picker a prefill pool rotates (the KV-aware picker orders it by overlap instead). The new "Live-fleet validation" section of docs/en/architecture/dynamo-integration.md records what was tested (Dynamo 1.4.2, kubernetes pod-mode discovery, vLLM 0.25.0 aggregated workers, a --router-mode direct --dyn-chat-processor vllm frontend, the kubernetes_pods source), the divergences (the vLLM chat processor dropping the worker pin on unpatched 1.4.2, multi-pool namespaces in #1888, nil ActiveLoad fields, OS-assigned publisher ports), and that a dynamo.vllm prefill/decode pair with NIXL and SGLang's concurrent bootstrap path remain untested.

• Enforce the dynamo section's field rules on the load and hot-reload path (#1869). Until now validate_config never ran the section's Validate, so its documented rules were checked by config validate only, and a file that broke them still started the router. They now apply at load, on every hot reload, and on an Admin API import, whether or not dynamo.enabled is set: the shape of each discovery.etcd.endpoints entry (no embedded credentials, query, fragment, or path; a dialable scheme or a bare host:port), at most 16 entries with no http:///https:// mixing and no tls beside an http:// entry, a non-empty discovery.etcd.namespace with no /, whitespace, or control characters, a non-empty auth.username, parseable watch_reconnect_backoff durations with max >= initial, the load and shim rules, endpoints when the etcd source is enabled, and the new discovery.kubernetes block whenever it is present. Upgrade note: run continuum-router config validate on the file before upgrading; a dynamo stub that fails it, enabled or not, now stops the router at startup with an error naming the field. In the same change continuum_dynamo_discovery_connected gains a source label (etcd or kubernetes_pods) and only the configured source is exported, at 0 from startup until its watcher connects; a hot source switch replaces the series. An alert on continuum_dynamo_discovery_connected == 0 therefore keeps working unchanged, and a dashboard that selected the unlabelled series should add max by () or a source matcher. The Pod source keeps the gauge at 1 across the server's routine 290s watch close and the relist that follows; it drops to 0 on a failed pass, a watch that ends with an error, or a stop. A watch silent for 320s (a half-open connection), or closed well before the 290s server timeout (a swallowed transport EOF), ends as an error, so the gauge drops and a relist follows, and the list is read in pages of 500 that stop at the retention ceiling.

• Record a refused input mask as the block the client received, under the new category guardrail_mask_unappliable (#1837, epic #1796). Client-visible: a masking verdict the router cannot apply (a redaction across two strings, more changed strings than the re-check budget, a rewrite that does not land, a masked request a provider still wants to change, a provider failure in a masking pass, or a masked request the ingress cannot turn back into its typed form) now answers with x-guardrail-category: guardrail_mask_unappliable and that category in the block body, instead of the generic guardrail, which remains only on the terminal chunk of a stream the output gate cut. Alerts keyed on guardrail for masking refusals must add the new value. The request-level decision is now recorded once the outcome is final, so guardrail_verdicts_total and the audit decision record count such a request as one block (and a masking pass that outran its deadline as one block under guardrail_timeout) rather than the transform the first pass used to record; the typed round trips of /anthropic/v1/messages (its thinking gate included) and /v1/responses run inside the masking core through the new gate_input_document_accepting and gate_input_document_deferred_accepting entry points, so a refused round trip is the decision recorded, or folded into the request's pending decision (#1836), under guardrail_mask_unappliable, and /v1/completions folds its write-back refusal into the request's aggregate, so each of these paths records exactly one decision. Monitor mode never masks, so it records transform for traffic enforce mode refuses as guardrail_mask_unappliable. The masking core records its decision under the policy it read before its first pass, so a reload during the masking passes neither drops that decision nor changes its mode label; a pending decision (#1836) records under the policy its kept pass read before its first provider call too (#1847). The tool-definition gate folds its accumulated verdict into an uninspectable refusal, so a monitor-mode deny followed by a tool that overflows the leaf cap is reported with the deny. On a Continuum Hub router the heartbeat verdict counters move from transform to block for these requests; the category itself never reaches the hub, which counts only provider blocks by category (both guardrail and the new value fold into other).

• Read built-in pii phone candidates in context (#1837, epic #1796). The phone pattern had no word boundaries and only a 7 to 15 digit count, so it masked IPv4 addresses, epoch timestamps, compact dates, process ids, file sizes, and digit runs inside hex hashes and UUIDs, which agent traffic resends every turn and which, since #1827, refuse the whole request when they sit in a tool contract field. A candidate must now not touch a letter, digit, _, /, or a . that follows a digit, must not be followed by a . or a - and a digit, must not be a dotted quad, must use dots only in the phone form (at least two, groups of at most 4 digits, so decimals and coordinates do not match), must not have a digit group longer than 5 before its last one, must not be hyphen-joined to a token whose only letters are hex digits (the all-digit groups of a UUID or hash; a plain word like phone- or -ext does not block it), and a bare digit run counts only in the trunk-prefix form (a leading 0 and 9 to 11 digits, such as 01012345678); trailing spaces, dots, and hyphens are no longer part of the masked span, and a rejected match no longer hides a number that starts inside it. Trade-off: an unformatted number without a trunk prefix, such as 4155550123, is no longer detected, while its formatted forms (415-555-0123, 415.555.0123, (415) 555-0123, +1 415-555-0123), including one hyphen-joined to a word such as phone-415-555-0123 or Tel-010-1234-5678, are; configure an external recognizer when bare numbers matter.

• Re-vendor the protocol crate from a single Continuum Hub revision (#1794, Phase 2 of epic #1796). crates/continuum-protocol/src and tests are now a byte-identical copy of hub rev 92907b47403b325b2a7038ad1f9ecf4ba4d07fbf, named on one provenance line in its Cargo.toml in place of the former local-ahead notes, and the router uses the hub's definitions everywhere: GuardrailPolicyMode and GuardrailBlockBehavior replace the router's statement enums, the smart routing, guardrail provider content, provider inventory, ttft_absent_reason, routing_enabled, and invalid_model_budget_usage_watermark surfaces come from the hub unchanged, and the wire fixtures are the hub's. Router-only helpers (the guardrail summary emission bounds and category folding, the threshold conversion) moved under src/control_plane/. The guardrail apply path now validates through the hub's acceptance entry point, so the receive bounds are unchanged (64 category entries, 256-byte category ids checked only for emptiness and length, 64 route entries, 256-byte route keys, and an unknown block behavior still tolerated), but four things every emitting hub already satisfies are now required: a guardrail statement without a digest is refused as digest_mismatch instead of being applied on the router's recomputation, and a policy without a mode, with an empty stage selection, or with a control character in a route key is refused as invalid_guardrail_policy. Each refusal is scoped to the guardrail track: the rest of the envelope applies and the held guardrail policy stays in force. On the same receive side, EquivalenceClass.members is now required on the wire, as the hub declares it; every hub since v0.5.0 always emits it. The router leaves the hub-ahead usage fields it does not stamp (UsageRecord.project_id and the web-search metering fields) absent, and apply clears any PolicyEnvelope.search_credentials and search_credentials_digest a hub sends, since this router never advertises search_credentials_v1 and holding them would leave plaintext per-user search secrets in the held snapshot; usage records now always carry compressed, and PROTOCOL_VERSION stays 0.

• Adopt Continuum Hub's definitions for the batch lifecycle, TTFT measurement source, model-budget, and optimization policy wire types in the vendored protocol crate (#1793, Phase 1 of epic #1796), copied verbatim from hub rev e6aae11e. The router now names its per-transition batch report BatchStatusTransition (state enum BatchJobStatus) and still posts that flat body to /api/agent/v1/batches. Usage records write the hub's canonical ttft_source key instead of ttft_kind; the hub reads both, so older and newer hubs keep ingesting TTFT, and ttft_absent_reason is unchanged. Model-budget types move into policy.rs under the hub's names (ModelBudgetTierPolicy, ModelBudgetEntry with the hub's sell_spend_microusd, ModelBudgetDimensionStatusWire), digests are unchanged, enforcement still reads only the hub's sell-spend verdict, and a statement whose tiers all have no pools is now an authoritative clear, matching the hub. The optimization block, semantic_similarity_bps, and cache_ttl_secs are optional as on the hub: an absent block or value keeps the router default, a TTL beyond u32 seconds saturates, and the Admin policy view reports optimization: null when the hub sent no block and omits semantic_similarity_bps and cache_ttl_secs inside the block when the hub did not state them (previously 0); the router uses its default. Header constants take the hub's *_CONTROL_HEADER names with unchanged values. PROTOCOL_VERSION stays 0 and the wire is unchanged.

• Judge a non-streaming chat reply once when fallback.quality_check and usage.quality_judgment share a target (#1802). With the same target block and equal max_request_bytes / max_reply_bytes, and no tool results sent to the usage judgment, the quality check's call over the original reply also asks language_match as a best-effort question that can never fail the check or change its verdict, and the usage record reuses the answers without a second decision call or a max_in_flight slot; a question the shared call left unanswered is asked alone and merged, and a merge across two resolved judge models is dropped as invalid. UsageQualityPlan::finish_text_with is the new wire-neutral core behind finish and finish_with. The shared call counts under systemone_requests_total{consumer="fallback_quality"}. When the check made no judgment, the usage judgment calls on its own as before; different targets or caps keep two calls. The /v1/decisions endpoint's per-target client cache now uses the shared ClientCache (capacity 256, keyed by target name), and a failed target build still warns once.

• Answer an input guardrail block before a 503 smart_routing_rejected load rejection and before a per-key 403 on the routed model, on a smart-routed /v1/chat/completions request whose classifier leaves the router (#1816). The input stage now runs ahead of classification there, so a blocked prompt gets the block response even while load management or the key's model allow-list would refuse it.

• Answer the smart routing Admin endpoints as disabled whenever the live switch is off (#1711). On a router that started with smart routing enabled and had it disabled at runtime, GET and PUT /admin/smart-routing/policies, POST /admin/smart-routing/classify, POST /admin/smart-routing/simulate, and /admin/smart-routing/cache/* now answer 400 (they used to answer 200 from the switched-off router), and GET /admin/smart-routing/stats reports load_snapshot: null while load management is off.

• Build the smart router and load monitor at startup even when smart routing is disabled (#1711). Both start idle, so enabling smart routing or load_management in a reload, or in a hub-composed configuration, now takes effect without a restart; before, a router started with smart routing off ignored a later enable. A disabled router costs a request one atomic load (the load monitor's enabled switch is now an AtomicBool), and the load assessment task starts the first time load management is enabled.

• Answer a hub-governed Admin write that has no configuration channel with a typed 503 (#1711). PATCH/PUT on the guardrail endpoints and PUT /admin/smart-routing/policies / model-profiles return {"error": {"code": "hub_governed_no_config_channel", "section": ...}} (the guardrail body keeps its status and message fields); the smart routing endpoints used to answer 200 updated while the edit was dropped.

• Deprecate the per-component smart-routing fields of UpdatableHandles (tier_registry, smart_router, load_monitor, extensions_source) in favor of smart_routing: Option<SmartRoutingRuntime> (#1708). The fields still work: when smart_routing is unset, HotReloadService::new builds the runtime from them.

• Emit the action-safety adapter's safeguard_results in the wire shape captured from a real Claude Code session (#1694). Conversion-backend responses and the streaming safeguard_results event now carry a top-level array with one dangerous_tool_use check whose status.tool_uses map is keyed by tool-use id: allow reads {"type": "evaluated", "outcome": "not_flagged"}, identical to what a native backend returns, and deny reads outcome: "flagged". When any proposed action resolves indeterminate (timeout, unreachable evaluator, budget exhaustion, and the other fail-closed cases), or cannot be correlated by id, the whole check reads status: {"type": "unavailable"} with no map. The old {"results": [{"verdict", "reason"}]} object is gone, and the evaluator's reason no longer reaches the client; it is logged at debug level. The deny, unavailable, and streaming mappings are unobserved on a real client and documented as decisions in the safeguards design note.

• Reload the Dynamo etcd connection identity (dynamo.discovery.etcd.endpoints, tls, and auth) live instead of rejecting a change as restart-required (#1756). The discovery watcher already built its etcd client on every pass, so the boot pin only turned a working change into an error: the discovery supervisor now replaces the watcher, empties the registry, and the replacement dials the new cluster with a full resync. Nothing in the dynamo section needs a restart any more. pin_boot_dynamo_connection and assert_dynamo_connection_unchanged remain as deprecated no-ops.

• Mark continuum_router::services::disaggregated::BackendInfo #[non_exhaustive] (#1759). The struct gained fields in #1698 (in_flight became u64, waiting_requests and kv_cache_usage were added), which already broke struct-literal construction outside the crate; code outside the crate now builds one with BackendInfo::new and sets the public fields afterwards, so a later field no longer breaks it again.

• Relay a client error on the buffered disaggregated arm instead of falling back (#1759). A 4xx the routed backend (or a gate in front of it) answered, other than 408 and 429, is now the request's answer on the non-streaming path too, stamped with x-continuum-routing-path and counted once on the routed path, exactly as the streaming arm has done since #1752; it is no longer a decode error and is no longer re-sent to unified serving under fallback_to_unified.

• Count a Dynamo aggregated fallback as continuum_dynamo_selection_total{role="prefill", reason="no_prefill_pool"} and keep the decode worker's own reason (#1759). The picker used to overwrite the decode reason with no_prefill_pool, losing which comparison placed the worker; the {role="decode", reason="no_prefill_pool"} series no longer moves.

• Tighten Dynamo discovery identity rules (#1759). A card worker_type this build does not recognize is now no claim beside a prefill or decode component, so the component decides instead of the worker being excluded as a mismatch, which an upstream rename of the spelling would have done to a whole fleet. A legacy card (empty component and endpoint) and a scoped base card for the same instance are refused together and reported as kind="duplicate", instead of the scoped card silently displacing the legacy one. dynamo.discovery.etcd.namespace now also rejects control characters.

• Spell served model ids verbatim in the per-model metric labels (#1879). model_usage_total, model_tokens_processed, llm_tokens_total, and the smart routing families (source_model and target_model on smart_routing_decisions_total and smart_routing_requests_total, model on smart_routing_model_availability) now spell a model id with the rule the fallback families already use: case is preserved, / and : are kept beside Unicode letters, digits, -, _, and ., any other character becomes _, the value is truncated at 128 bytes on a character boundary, and an empty id becomes unknown. Previously meta-llama/Llama-3.1-8B-Instruct exported as meta-llamallama-3.1-8b-instruct and Qwen/Qwen3-8B and qwen/qwen3-8b shared one series, while smart routing only truncated and let whitespace and control characters through. Series identity changes for every model whose id carries uppercase, /, or : (on the smart routing families also whitespace or other punctuation): the old series stop and new ones start, so dashboards and alerts that match a lowercased or slash-stripped spelling must be updated, and /admin/metrics/history keeps samples recorded before the upgrade under the old spelling. The model cardinality guard now keys on the exported spelling, so distinct models that used to collapse into one tracked value count separately against its fixed limit of 1000 models. Non-model labels (api_key_id, backend, guardrail provider and category, type, and the smart routing policy_name) keep their spelling. GET /admin/config/schema also stops serving a pattern on the fallback.fallback_chains key and entry nodes: its \p{...} escapes are valid in ECMA-262 only with the u flag, so Python jsonschema and several Go and Java validators rejected or mis-parsed the schema. Both nodes keep minLength: 1 and maxLength: 128, the description still names the accepted characters, and the server-side rule is unchanged.

• Wire metrics.max_model_labels and deprecate the unread metrics limits (#1897). max_model_labels now sets the model cap of model_usage_total, model_tokens_processed, and llm_tokens_total; it was parsed but never read, and the cap was a compiled 1000. Its default moves from 100 to 1000, so a deployment that never set it keeps the same effective cap, and a configured value takes effect at startup (restart to change). Lowering it evicts nothing: tracked models keep exporting and new ones are refused. metrics.max_backend_labels and the metrics.cardinality_limits block remain ignored, still parse so older files load, and now draw a config validate and load warning when set to a non-default value. The cardinality guard also accepts an already tracked label value under a read lock instead of taking the write lock on every request.

• Build portable binaries from a source checkout by default (#1799, PR #1810). .cargo/config.toml no longer sets -C target-cpu=native for x86_64 Linux and macOS, which also applied to host proc-macros and build scripts and could crash a build or cached test binary with SIGILL when artifacts moved between CPU generations. Opt back in locally with RUSTFLAGS or a user-level ~/.cargo/config.toml.
• Shrink the Dynamo load subscriber's resident buffers from about 48 KiB to 2,624 bytes per idle subscription (#1755, PR #1764). The ZMTP frame buffer starts at 512 bytes and grows by powers of two only after the declared length passes the MAX_FRAME_BYTES check, and the socket buffers are 2 KiB read and 64 bytes write instead of tokio's 8 KiB defaults.
• Make cargo deny check cover every optional feature graph and pin the protobuf compiler CI installs (#1757, PR #1760). deny.toml sets all-features = true, so a license or advisory problem in a dynamo, control-plane, appproxy-router, or otel dependency fails the check, and every arduino/setup-protoc step (CI, release, and the weekly dependency float, which now installs it) pins version: "36.2".

Fixed¶

• Keep routing-policy conditions the WebUI policy editor cannot edit when saving from the Smart Routing page (#1988, PR #1990). The editor rebuilt each when from complexity, domain, and requires only, so saving sent a policy narrowed by key_tier, org, language, header, or input_tokens back as when: {}, where it matched every request. Those conditions are now carried through unchanged and listed on the policy row.
• Keep reserved model-experiment variants out of the smart-routing status and simulate, and keep smart_routing_model_availability in step with the candidate pool (#1989). A variant model of a hide_variant_models: true experiment was filtered from the request path only, so GET /admin/smart-routing/status listed it under candidate_models (or excluded_unprofiled_models) and POST /admin/smart-routing/simulate could return it as the target although no request routes there; the candidate pool now drops reserved variants itself, and the request path, status, and simulate read that one pool. The availability gauge is rebuilt on every collection instead of being written one model at a time: a routable candidate is 1, a model that is unhealthy or excluded as unprofiled under unprofiled_models: exclude is 0 (an excluded model used to read 1), and the series of a model that left the pool (a backend turned internal: true or disabled, a newly reserved variant, a model gone from the aggregation) or moved to another tier is removed rather than kept at its last value until restart. A failed model aggregation leaves the gauge unchanged. SmartRoutingMetrics::set_model_availability is replaced by SmartRoutingMetrics::refresh_model_availability, which takes the whole collection and its sequence number, so a collection that finishes after a newer one is not applied.
• State the share of requests hybrid actually escalates, and make it observable (#1973). The docs promised that hybrid mode escalates only genuinely ambiguous requests, "typically 10-20% of traffic", but the rule confidence counts signals instead of estimating whether the label is right: on the labelled accuracy dataset hybrid escalates 53.7% of requests at the default rule.confidence_threshold of 0.7, on public benchmark prompts 91% to 99% fell below the threshold, and the confidence ranks a correct complexity label above an incorrect one no better than chance (AUROC 0.46), so no threshold selects the ambiguous minority. The configuration docs now give the measured shares, say what the score is, and name rule with shadow: true or hybrid_system_one as the alternatives where the share is high. A new smart_routing_hybrid_gate_total{method, outcome} counter (method hybrid or hybrid_system_one, outcome routed or escalated) makes the share readable from /metrics, and the accuracy harness now reports the confidence of correct and incorrect labels and the escalation share per mode. The rule confidence itself and every default are unchanged.
• Accept registry pull Secrets in the continuum-event-broker Helm chart in both list forms (#1984). imagePullSecrets entries may now be - name: <secret> or the bare - <secret>, both rendered as {name: ...} on every broker Pod, and the render fails on an entry of another shape, an invalid Secret name, or a Secret listed twice instead of producing a Pod spec the API server rejects. The chart README, the example values, the Kustomize bundle (as a patch), and the deployment guide now document pulling the broker image from a private registry or mirror; the chart still never renders the credential.
• Classify a translation request as multilingual with the rule classifier (#1976). DomainTag::Multilingual was documented as translation, and the LLM and System One classifiers answer it for a translation request, but the rule classifier set it only from the share of non-ASCII characters and suppressed it for primary_language, so an English translation request read general, as did a Korean one on a Korean deployment, and when.domain: [multilingual] never matched either. A new translation_intent domain signal fires on translate as a word (so "translation lookaside buffer", CSS translateX, and "untranslated" stay out), 번역 (not 번역가, 번역자, 번역기, 번역본, or 번역서), 翻訳, 翻译, and 翻譯, maps to multilingual, and is not suppressed by primary_language; at 0.72 it outranks a single creative or a two-marker analysis match and yields to fenced code, code_intent, and math. Like other intent signals it counts toward rule confidence, so a short translation request now clears the default 0.7 hybrid gate instead of escalating. The script check remains the fallback when no intent signal fires. The type doc and the domain and signal tables in the configuration docs now state this meaning, and the three translation cases in the accuracy dataset (and the two router-derived rows of the System One routing corpus) are relabelled multilingual; domain accuracy stays at 100%. The calibration derived.json files were re-derived offline from the committed raw runs (the rule domain accuracy on the routing corpus moves from 0.445 to 0.440 and rule alone from 0.477 to 0.475, because the committed runs recorded two router-derived translation rows as general; against the corrected corpus labels both classifiers gain those rows), and every derived System One default is unchanged either way.
• Read a Korean request at the same complexity as its English translation (#1972). The rule classifier's Korean simple keywords were bare question words where the English ones are request phrases: 무엇 matched every "what" and "which" question, 어떻게 every "how", 설명 the noun and the descriptive verb, and 정의 the genitive of any noun ending in 정 (과정의), so a Korean request read a band lower than its English twin far more often than the reverse (212 of 251 disagreements on 809 benchmark pairs). 설명 now counts only in its request forms (설명해, 설명하세요, 설명 부탁), 무엇 only as "what is" (not …것은 무엇입니까, 무엇이었는가, 무엇을, or the pronoun 무엇인가를), 어떻게 only as "how do I" (not 어떻게 바꾸었는가, 어떻게 되었나요, or 어떻게 지내세요), and 정의 not across a genitive unless a verb ending follows. The noun 증명 no longer counts as a complex keyword on the score path, only its verb forms (the English entries, prove included, are unchanged). A Hangul syllable or CJK ideograph now weighs two and a half Latin characters in the length estimate instead of two: on the repository's 36 paired English/Korean fixtures the Korean members estimated 12.9% shorter before and 1.6% longer after. On the accuracy harness Korean complexity goes from 24/34 to 25/34 on the original cases with English unchanged at 27/34, 30 of 32 pairs now read the same band (27 before), and the complexity gate rises from 72% to 73%. The Korean greeting 안녕하세요! 오늘 하루 어떻게 지내세요? now estimates 11 tokens and reads simple. The System One calibration derived.json files were re-derived offline from the committed raw runs, since they score the rule classifier beside System One: the rule figures in the routing shadow table moved (rule alone 0.472 to 0.477 on all), every derived default is unchanged, and the non_latin slice no longer has a floor that beats the rule classifier.
• Keep models served only by internal backends out of the smart-routing candidate pool (#1975). Smart routing chose among every healthy aggregated model, so a model whose only backend was internal: true (or enabled: false) could be picked for a user request. The pool now applies the model listings' visibility rule: such a model is never a target, and a model that also has a public backend is health-checked on its public backends only. An internal backend can still serve the LLM classifier.
• Release the per-user web-search credential binding when a reload removes web_search, and meter every search path (#1970). The binding that advertises search_credentials_v1 and lets the store hold Hub-delivered search secrets was latched once and never released, so after a reload removed the web_search section the router kept advertising the capability, acknowledging search_credentials_digest, and holding personal secrets no request could use. It now follows the section on every configuration revision (startup, file reload, and Admin apply): releasing it withdraws the capability and drops the held secrets and the acknowledged digest at once. The Anthropic web_search_20250305 emulation (Claude Code's WebSearch against self-hosted backends) now records its query-extraction turn in the statistics and the control-plane usage record, with the search metering, and records its early failures; each search is reported in exactly one record, even across the attempts of a fallback chain; a chat completion or Anthropic tool loop whose later turn fails after searching now emits one usage record with zero tokens and the metered searches instead of dropping searches a member's key paid for. A failed request that made no search still emits nothing. The local statistics row of a failed chat completion now names the backend that returned the failure, where it used to say unknown.
• Keep the multilingual script signal out of the rule classifier's confidence (#1971). With no classifier.rule.primary_language, a short trivial Korean question such as 대한민국의 수도는 어디야? scored exactly the shipped rule.confidence_threshold of 0.7, because the signal (strength 0.7) counted once as a signal and once as a strong one, so hybrid and hybrid_system_one routed on the rule result without asking the LLM or System One classifier, while the same question under primary_language: ko scored 0.5 and escalated. The signal still decides the domain when no intent signal fires; it no longer moves the confidence, so both configurations now score Korean text alike. The confidence sum is also snapped to a 1e-9 grid before clamping, so an exactly at-threshold combination (four signals with one strong is 0.8, not 0.7999999999999999) passes a non-default threshold, since every combination is now the f64 nearest its decimal value. method: rule labels are unchanged.
• Report the effective smart-routing state from GET /admin/smart-routing/status (#1974). Status read classifier_method, the policy count, and the virtual model from the local smart_routing: section, so under a Hub policy it could name a method the router was not running. The top-level fields now come from the live smart router and tier registry, which is a behavior change for any consumer that read them as local values; the local values move under a new configured block. New fields source (hub or local), hub_governed, smart_routing_policy_digest (the executed Hub digest, equal to the one acknowledged to the Hub), and declined_hub_policy show which policy is in force, and the WebUI status tab shows the source. POST /admin/smart-routing/simulate now returns the same classification object as POST /admin/smart-routing/classify (adding language, required_capabilities, reasoning, and signals), built by one serializer, and both responses are documented in the Admin API guide.
• State one context window per model on GET /v1/models/extended (#1479). The flattened upstream fields kept one backend's max_model_len (or an aggregator's context_length) verbatim beside the router's metadata.limits.context_window, so the object stated two different windows whenever they differed: when backends serving one id disagree and the router publishes their minimum (#1456), and when model-metadata.yaml or a model-metadata.d/ drop-in supplies limits. Either key, when present, now carries metadata.limits.context_window, the window the router honors; neither is added when the backend did not send it, neither is rewritten when the published window is 0 (not applicable), and a single-backend model whose window came from that key serializes exactly as before. Every other upstream key, including pricing and top_provider, still passes through verbatim, and metadata is the authoritative source for anything both state. GET /v1/models, GET /v1/models/{model}, and GET /admin/models are unchanged. The API reference gains a section for GET /v1/models/extended.
• Honor a declared reasoning_efforts vocabulary on the streaming Gemini route (#1668). A streamed /v1/chat/completions request to a gemini-typed backend is served by the typed GeminiBackend, whose Anthropic-native gate pins the backend type and reads no declaration, so a request carrying thinking or output_config was fitted to Gemini's default vocabulary when streamed and to the declared one when not. The streaming dispatch now runs that gate against the configuration snapshot the request was routed against before the typed transform, so the backend's model_configs and the shared metadata layer (a model-metadata.d/ drop-in) resolve the same way on both routes. A regression test also pins the mid-stream fallback relay, which already resolves each hop's declaration from the routed snapshot.
• Apply a runtime edit of the smart_routing section to the live smart router (#1715). A configuration publish whose only change was smart_routing, such as PUT /admin/config/smart_routing switching classifier.method from hybrid to llm, was reported by the config API but never reached the running classifier, because the hot reload diff did not compare the section and so applied nothing. The section is now compared and a smart-routing-only edit rebuilds the classifier. A model_experiments-only edit, which was already read per request, is now also reported as a change.
• Score the rule classifier on the current turn, so conversation history no longer moves a smart-routing decision (#1714). The classifier read the whole conversation: every message counted toward the length signal, a conversation_depth signal counted messages, and keywords and code markup from earlier turns fed the score, so a trivial question after two short exchanges read simple, after one complex exchange moderate, and the added signals lifted the rule confidence to the default 0.7 rule.confidence_threshold. hybrid then stopped escalating to the LLM classifier and routed the follow-up to a larger tier than the identical question asked with no history. Complexity, domain, and confidence are now computed from the system and developer messages plus the latest user message, the same text the LLM classifier judges, under every method (rule, hybrid, and the rule fallbacks of llm, system_one, and hybrid_system_one). The published language is now the latest user message's, so a Korean question after an English conversation matches a when.language: [ko] policy; it falls back to the language of every user turn only when that message has no letters. required_capabilities still reads the whole conversation, so an image in an earlier turn still requires a vision backend. The conversation_depth signal is removed. Single-turn requests classify exactly as before, and complexity accuracy on the labelled dataset is unchanged.
• Attribute non-streaming usage served through the Dynamo shim, and every other buffered disaggregated reply served by the selected backend, to the presented hub key (#1953, epic #1703). The buffered disaggregated path recorded its usage without the request headers, so the control-plane usage tap saw no API key, stamped the record key_id="unattributed", logged the fail-open warning, and charged no key: these requests did not count toward the key's TPM windows or monthly budget delta. The path now records with the client's headers and the same hub routing attribution and context-compression metadata the unified path records, so the record carries the matched key's key_id, user, and team, and the usage now counts toward the key's limits exactly once. Streaming attribution and local-cache replay metering are unchanged. The fail-open warning control-plane: usage recorded without hub key attribution now carries a cause field (no_policy_store, policy_not_synced, no_presented_token, unmatched_token, or no_stream_grant) and names the cause in its message without logging the token; only no_policy_store keeps the advice to set control_plane.policy.enabled: true. Each cause is rate-limited on its own one-minute window.

• Record fallback-chain usage under the model that served the reply (#1934). When fallback.fallback_chains moved a request off a failing primary, the chain target's tokens were recorded under the model the client asked for, beside the served backend and backend_id, so /admin/stats credited the primary with tokens it never produced and the control-plane usage record paired the served provider with the wrong model, missing the Hub model-budget and cost-center pool that was actually consumed. The /admin/stats model row, the per-model and per-key token counters, model-experiment token attribution, and the usage record's model now name the chain target's configured id on buffered /v1/chat/completions, /v1/completions, /v1/embeddings, rerank, and sparse embeddings, whatever notify_on_fallback says, and on streaming chat for a hop taken at selection, before the first byte, or mid-stream. A plain chain hop stamps no requested_model; under a Hub substitution, arbitrage, or budget fallback requested_model keeps the policy's value. Model-budget admission still checks the requested offering's pool while settlement charges the served one. Library surface: the public MidStreamFallbackContext::begin_provider_attempt takes the attempt's model as a second argument. /anthropic/v1/messages and /v1/responses chains are unchanged.

• Scope the Responses session store to the router instance instead of the process (#1941). Stored responses (store: true) lived in one process-wide store, so two routers in one process (serve_embedded, the library builder, or the members of one grouped test binary) read and overwrote each other's sessions whenever their upstreams answered the same id: a passthrough reply is stored as pt_<upstream id>, and the second write replaced the first. Each AppState now gets its own store on its first Responses request, with its own background cleanup task and its own 10,000-session cap; a dropped state's store and cleanup task are released the next time another state registers one. A process with one router, which is every production deployment, behaves as before, and hot reload keeps the same store.

• Rate-limit repeated backend-refusal log lines per model (#1926, epic #1703). With a model's only backend unhealthy, each refused request used to write about four ERROR or WARN lines (No healthy backend found for streaming model ..., Failed to select backend for model ..., Router error: All backends are unhealthy, and on the Dynamo shim path Disaggregated fast decode streaming dispatch failed), so a client retrying 21,600 times a minute produced about five million log lines in an hour and buried the health-check transitions that explained the outage. These sites, plus the non-streaming No healthy backends available from candidates, the retry loop's closing Exhausted all retry attempts ... and Non-retryable error ... lines for these refusals, and the per-backend rate-limit refusals, now log the first refusal per site and model at WARN, keep repeats within a 60-second window at DEBUG, and write one WARN summary with the count, the model, and the reason (N requests refused for model 'X' in the last 60s: all backends unhealthy) when the next refusal arrives after the window. The table of tracked models is bounded. Refusal metrics are unchanged. These refusal lines are written at WARN, not ERROR: an alert that matches ERROR-level Router error: All backends are unhealthy, Exhausted all retry attempts, or No healthy backend found lines should match the WARN level or alert on the refusal metrics instead.

• Stop the CI test step from silently skipping tests (#1485). cargo test --tests -- --skip integration_test never excluded tests/integration_test.rs, because libtest's --skip is a substring match over test paths and a target name is not part of them; it instead dropped the four tests in config_integration_tests (tests/mid_stream_fallback_test.rs) and metrics_integration_tests (tests/rate_limiting_test.rs). The filter is removed from .github/workflows/ci.yml and scripts/local-ci.sh, so those tests now run in CI, and the comments and step label no longer claim an exclusion.

• Accept HuggingFace and Ollama model ids in fallback.fallback_chains (#1585). A chain model name, as a primary or as an alternative, may now contain / and : next to the Unicode letters and digits, -, _, and . it always accepted, so ggml-org/gpt-oss-20b-MXFP4: [meta-llama/Llama-3.1-8B-Instruct, llama3:8b] validates and falls back like any other chain; the 128-byte cap is unchanged, whitespace, control characters, and other punctuation are still refused, and the name is used as exact bytes. A refusal now names the character and its byte offset, and repeats the refused value escaped and capped at 128 characters, so a control character, an ANSI sequence, or a bidi override cannot reach logs or the validate and Admin responses raw. fallback.quality_check.models shares the same rule. decisions.targets[].name and model experiment names and variant ids stay [A-Za-z0-9._-], since the operator chooses them, and a refused value that looks like org/model now says so. The fallback metrics (fallback_attempts_total, fallback_success_total, and the rest) now export / and : in original_model and fallback_model instead of mapping them to _, and truncate at 128 bytes on a character boundary instead of 64 bytes mid-character, which could panic on a long Unicode model name; a dashboard that matched the underscore spelling of a slash-bearing id sees a new series. The WebUI chain editor and the Admin configuration schema follow the new rule.

• Classify a short formal proof as expert under the rule classifier, and warn from config validate about a routing policy that depends on expert where the rule classifier decides it (#1617). The rule complexity score is a weighted average bucketed at 0.75, so a request under 200 estimated tokens without math notation cannot score above 0.683, and expert in practice meant long rather than hard. A request whose latest user message carries math notation and whole-word prove/proof (or 증명) now reads expert at any length, unless its simple keywords outnumber its complex ones; notation in the system prompt or an earlier turn does not count. A short novel-architecture or research-level request still reads complex or lower, because its signals are identical to a routine design request's; the Complexity Levels section of the advanced configuration guide gives the arithmetic, including the one other short route (51 to 200 estimated tokens with math notation and four or more distinct complex keywords). Under an operator-enabled reject_expert load state, short proof requests that used to read complex are now rejected. config validate warns at smart_routing.routing_policies[N].when.complexity about a policy that matches expert without complex under method: rule, hybrid, or hybrid_system_one (hybrid only while rule.confidence_threshold is at most 0.95). Rule-mode complexity accuracy on the labelled dataset rises from 69.5% to 76.3%, and the accuracy gate from 65% to 72%.

• Tighten the rule classifier's formal-proof floor, and warn from config validate when the LLM classifier has no backend (#1892, follow-up of #1617). The about-a-proof guard now compares simple and complex keywords on the latest user message, the same turn the math notation and proof wording come from, so a "what is" in an earlier turn no longer suppresses a proof request and an "implement" in an earlier turn no longer turns a question about a proof into expert. The Korean stem 증명 no longer matches after a syllable that closes another word ending in 증 (검증명령, 인증명세, 보증명세서), while unspaced proof compounds such as 존재증명 still do. method: llm or hybrid with no smart_routing.classifier.llm.backend pin and every backend internal: true or enabled: false now draws a warning at that path, and the expert-ceiling warning covers the same case. A blank pin is now read as unset at runtime, as the load check already read it, so it takes the first user-routable backend instead of silently dropping the classifier, and a pin padded with spaces resolves to the backend it names. config validate warns about a blank pin when the method is llm, hybrid, or hybrid_system_one, since an empty value usually means an unset environment variable. Complexity accuracy on the labelled dataset (76.3%) and the committed System One calibration runs are unchanged.
• Even out the rule classifier's formal-proof floor across languages, spacing, and conversation history (#1899, follow-up of #1892). The about-a-proof guard now counts the proof term once, and only as a request verb, in both languages, so "이 증명을 설명해줘 \(\sum ...\)" keeps its score like "Explain this proof of \(\sum ...\)" instead of reading expert, while "증명하고 각 단계를 설명해줘" still takes the floor; improve no longer counts as a complex keyword inside the guard. The guard also reads the keyword table for the latest user message's own language, so a Korean question about a proof after a long English turn is no longer floored. 증명 after a medical word ending in 증 (합병증명단, 후유증명칭, 통증명칭) is no longer proof wording, and a verb ending after the stem overrides the preceding-syllable exclusion (보이고증명하시오 now takes the floor). disprove/disproof, a sentence-opening "show that" (optionally after "please" or "now"), and the Korean nominalized claim before 을 보이/을 보여 (무리수임을 보여라, 성립함을 보여라, 무리수라는 것을 보이시오) are now proof wording; "demonstrate" is deliberately not. The Korean form reads only a nominalized predicate ending (임, 함, 됨, 짐, 음, 움, 큼) or a -는 것 clause, so a noun that merely ends in ㅁ (이 그림을 보여줘, 교점을 보여줘, 덧셈을 보여줘, 관심을 보이는) does not take the floor. One over-read is accepted and pinned by a test: "Show that step again with \(\frac{a}{b}\)." takes the floor. The accuracy dataset gains nine proof-floor cases, one multi-turn through a new optional context field; complexity accuracy on the 68 cases is 75.0% (51/68, 67.6% before the fix), and the 59 earlier cases are unchanged at 45/59.
• Mask the right text when an external Presidio-compatible recognizer reports a pii span after a multibyte character (#1862, follow-up of #1857). Presidio reports start/end as Unicode code points (Python str indices), and the provider read them as UTF-8 byte offsets, so any accented letter, Hangul, Han, or emoji before an entity shifted the mask: 서버 주소는 10.20.30.40 입니다 became 서버 <REDACTED:IP_ADDRESS>0.20.30.40 입니다, and a span that no longer fell on a character boundary was dropped, so John Smith in 담당자 홍길동 님, 연락은 John Smith에게 went out whole. The spans are now mapped from code points to bytes before overlap resolution, so the whole entity is masked and the surrounding text is kept; ASCII-only text is unchanged. Offsets must be code points, so an analyzer that reports UTF-8 byte or UTF-16 offsets is not compatible. A reversed span, or one that runs past the text (what such an analyzer produces), is now a malformed analyzer answer that follows on_error: fail_closed fails the check as an invalid response body does, and fail_open masks the remaining valid spans, logs a warning with the count, and increments guardrail_degraded_total{kind="external_invalid_span"}; an empty span is still ignored. The built-in scanners do not detect IP addresses, person names, or IBANs, so this recognizer is the only path that masks them.
• Mask phone numbers that sit next to another number (#1863, follow-up of #1860). A phone number followed by a 5- to 12-digit run no longer leaks when the pattern's last group takes the run's first digits: the number is masked without that group when it starts with +, (, or 0, or when no later candidate inside it takes the rest of the run, so 415-555-2770 337543, +82-10-1234-5678 482913, and +81 3 8758 4450 082985 mask the phone number and 1588-8099 01099843848 masks both numbers, while order 1234 030 12345678 still masks 030 12345678. A match that starts one group early now also moves to a number whose next group has 5 or more digits when the scanner accepts that number whole (room 1419 07700 900159 masks the UK mobile, not 1419 07700), its leading group may have 5 digits (ref 12345 010 1234 5678), and a 4- or 5-digit token starting with 0 before a whole number, such as a Korean zip code, no longer leaves that number's last group in clear (06236 010 1234 5678 masks 010 1234 5678, 0163 020 7946 4552 is masked whole). A country code the 15-digit cap refused because the pattern took the next number's first group is masked again, the group given back (a number that starts with its own + keeps its last group, so room 0987 +86 138 0979 3831 still masks the whole number): +44 20 7946 5121 0630 masks +44 20 7946 5121, and +44 20 7946 0958 4111 1111 1111 1111 masks the phone number and the card, closing the known gap pinned in #1857. On a 240,000-line mixed corpus the phone digits left in clear drop from 28,878 to 190 and the non-PII digits masked drop by 10%, and on that corpus, the issue's shapes, and a 200,000-line adversarial corpus no phone or card digit that was masked before is unmasked.
• Mask a phone number written right after a card number whole (#1868, follow-up of #1863). The phone pattern could match from the card's last group, and the phone validator then read the card's digits as the number's leading groups, so the span was cut at the card's end and the number's last group went out in clear: 4681 3716 0697 9774 010 1841 1335 became <credit_card> <P> 1335. When a phone span starts inside a card and ends after it, the phone scan now runs a second time starting the candidate right after the card, and the detections of both passes are kept, so the number is masked whole and no digit the first pass masked is unmasked; text without such a span is scanned once. A number after a 1- to 3-digit token and a 3-digit token is also masked with both tokens (2 118 07700 900907, 8 681 020 7946 0402), where its last group used to stay in clear. A phone number whose first group the card scanner reads as a card's short last group or as the remainder of an overlapping card candidate keeps that group masked when credit_card: allow removes the card span (4545 9033 0689 8841 415 555 1780 masks 415 555 1780). On the 240,000-line mixed corpus of #1863 the phone digits left in clear drop from 190 in 40 lines to none and the non-PII digits masked grow by 0.21%. On that corpus, two 200,000-line adversarial corpora, two 60,000-line mixed corpora, and three shape corpora, no digit that was masked before is unmasked under the default mask policy, phone: block, phone: allow, or credit_card: block, and no phone digit is unmasked under credit_card: allow.
• Close the remaining phone leak shapes of the built-in pii provider (#1894, follow-up of #1868). A number whose match started one group early, on a group the left extension then moved back from onto the previous number's 0-led last group, kept that early start and lost its last group: +44 20 7946 0527 3700 020 7946 7480 became <P> <P> 7480, and 020 7946 5629 4279 3413 0472 1495 020 7946 2290, page 0 09305 020 7946 1556, 020 7946 0757 21087 07700 900578, and +82 10 1389 0551 5088 07700 900675 leaked the same way. A number after two short tokens either of which starts with 0 lost its last group too (177 021 07700 900011, 080 832 010 0689 4644, +44 20 7946 9625 071 962 020 7946 3640). When the first phone detection sees such a span, a second detection runs with a validator that retries the lead from the pattern's own start and accepts a 0-led two-token lead, and its spans replace a first-detection span only when they cover every digit of it, so no digit masked before is revealed. A card span that starts with 0 is no card number, so a phone number that starts it and ends inside it is no longer dropped with it, unless a real card starts at one of the span's later groups (0310 3782 822463 10005 stays allowed whole): with credit_card: allow, 07700 900700 79271 020 7946 7001 masks both phone numbers, and phone: block blocks it. A phone number inside a card-shaped false positive that does not start with 0 is still revealed under credit_card: allow, since no rule separates it from a real card, and the guardrails guide lists it as a known leak. On a 300,000-line ground-truth corpus (phones in seven national forms, cards in every accepted grouping, dates, times, prices, ids, versions, IP addresses, and adversarial adjacencies), a second seed of it, and a 200,000-line corpus in the distribution of the issue's prototype, no digit that was masked before is unmasked under the default mask policy, credit_card: allow, phone: block, phone: allow, or credit_card: block, and the non-PII digits masked grow by at most 0.08% under the default policy and 0.84% under credit_card: allow. On the prototype-style corpus the phone digits left in clear drop from 28,450 to 8 under the default policy and from 62,402 to 21,499 under credit_card: allow.

• Stop reading a short token and a real card's first groups as one phone number in the built-in pii provider (#1937, follow-up of #1894). A phone span that started on a token before a space-grouped card and ended inside the card was kept, since it did not lie inside the card: with credit_card: allow, 0311 3782 822463 10005 masked 0311 3782 as a phone number, and phone: block blocked it although it holds no phone number; 0003 4111 1111 1111 1111, 0311 3056 930902 5904, and 03 3782 822463 10005 did the same, and 0311 4111 1111 1111 1111, whose 0-led span passes the Luhn checksum, masked the card's last two groups. Such a span is now cut just before the card and dropped when fewer than 7 digits are left, and a phone span wholly inside the real card after a stray 0-led token is dropped, so these rows are allowed whole with credit_card: allow and masked rather than blocked with phone: block, while 02 1234 5675 4111 1111 1111 1111 keeps 02 1234 5675 masked and blocking. Both rules act only on a card written as its network prints it that ends its run of digit groups and that no other group of the run could start, and never cut into digits that could end a whole phone number, since the same text is as often a phone number followed by short numbers (02 1234 5678 9012 3452); 5678 4111 1111 1111 1111 and (03) 4111 1111 1111 1111 read that way too and keep their phone span. The second rule drops only a span whose digits card spans cover (a card glued to a letter, as in 0647 6515 4450 8558 3827 006ID 2076, is no card span) and that starts past any whole phone number the token begins (016977 4050 6177 9526 5756 826 keeps 4050 masked), and a number that starts without 0 or + may reach 15 digits, so a country code written without + (86 138 1234 4567 8901 2345 6783) keeps its digits masked. On 1.5 million ground-truth lines (the three #1894 corpora, 400,000 lines of real cards after 0-led, other, and parenthesized tokens and after phone numbers, and 300,000 lines of phone numbers followed by short numbers) and an independent 500,000-line corpus, no digit masked before is unmasked under the default mask policy, credit_card: allow, phone: block, phone: allow, or credit_card: block, and no non-PII digit is newly masked. Under credit_card: allow the card digits masked as a phone number fall from 2,512,026 to 2,262,464 and from 1,018,884 to 971,688. Under phone: block no line with a detected phone number stops blocking; 9 and 161 lines stop blocking whose phone number neither version detects (for example +33 1 74 53 03 18), where only the span over the token and the card's head blocked, a count that grows with the share of undetected phone formats.

• Detect phone numbers written in European, Russian, irregular national, slash-separated, and ungrouped formats in the built-in pii provider (#1947, follow-up of #1937). The phone pattern needs a 3-digit group followed by a 3- or 4-digit group, so numbers in 1- and 2-digit groups passed unmasked under the default policy and did not block under phone: block (+33 1 74 53 03 18, +7 495 301-77-50, 8 (495) 301-77-50), a 3-2-2 tail left its last pair in clear (+7 495 301 77 50, 044 668 18 00, 0212 123 45 67), and 01267-1-5398, 1800-87-6141, 030/12345678, and Mobile: 13800138000 were never detected. A second phone scanner now reads these forms, each with a strong anchor: a + or 00 country code of a country that writes its numbers in pairs, a trunk 0, Russia's trunk 8 in its exact shape, the toll-free 1800, or, for a French national number, a space-separated UK irregular number, and an unseparated Chinese mobile number, a phone word before it (Tél. : 01 42 68 53 00, Tel: 01267 1 5398, 手机13800138000). A generic phone word (home, work, office, ext, contact, call) anchors only right before the number or as call me at, and a strong one allows at most one other non-ASCII character between, so /home used 15393645540 bytes and 手机销量 16836428018 台 stay in clear. A national form after an identifier word (ref, lot, order, Kundennummer, 주문, 订单, and similar) or joined by a space to another number is not masked unless a phone word comes right before it, nor is a French-shaped sequence or lottery draw, a German zero-padded counter or file name without a phone word, or a pair-grouped data row whose code is no pair-writing country (+10 20 30 40 50). Known leaks stay pinned in tests and documented: a country code written without + or 00 (33 9 61 83 82 88), and a bare French national, space-separated UK irregular, or Chinese mobile number with no phone word before it (01 42 68 53 00, 01267 1 5398, kim,13800138000,seoul), whose shapes zero-padded table rows, dotted build numbers, unrelated adjacent numbers, and order ids share; the guardrails guide also lists the non-phone shapes the forms still mask (Step 03630/100000, Δ +43 79 06 04 33, 1800 12 2026, KW 05 123 45 67, home: 16235080111), which phone: block blocks. The card network table that gates the #1937 rules also gains Diners Club 16-digit and 3095, Mir, Maestro, RuPay, Elo, Hipercard, Verve, Troy, and the 13-digit Visa 4-4-4-1 grouping, so 0003 3056 9300 0902 0004 and 0311 3095 123456 7897 pass whole with credit_card: allow; a trunk-prefixed token before a 4-4-4-4 card (0312 4111 1111 1111 1111) stays over-masked by design and is now documented. On 500,000 ground-truth lines of the new formats in context and of false-positive risks, the 1.5 million lines of the #1937 corpora, and an independent reviewer's 830,000 lines, no phone or card digit masked before is unmasked under the default mask policy, credit_card: allow, phone: allow, or credit_card: block, and no detected phone number stops blocking under phone: block. Phone digits leaked in the new formats (the pinned leaks aside) fall from 1,901,810 to 83,135, and no line without a real phone number newly blocks on those 500,000 lines. The wider network table does make phone: block stop blocking 69 of those lines and 21 of the reviewer's whose phone number is in a form neither version detects (0003 6362 3386 3784 5325 tel 33 2 67 00 22 73), since the old code blocked them only through its span over a stray token and a card's head. The new forms mask 271 and 5 non-PII digits the old code left in clear (0.045% and 0.0002% of the non-PII digits masked before), while the card table releases 10,142 and 553 token digits. Detection takes up to about twice as long on dense numeric text, and about 20% longer on the regression corpora.

• Keep prefill and decode workers out of the unified fallback of disaggregated serving (#1864, follow-up of #1855). With fallback_to_unified on, a request disaggregated serving handed back (every cold request in a deployment with both prefill and decode workers, since the prefill-then-decode wire protocol is unavailable, and every failed fast-decode dispatch) was re-dispatched over every backend serving the model, so the default round-robin rotated onto prefill-only workers and back onto the decode worker that had just failed, contradicting the documented fallback to a unified backend. Both arms, buffered and streaming, now run that fallback with an allow-list of the caller's user-routable non-role backends minus the failed backend, used only while one of them serves the model and could be selected right now (health, circuit, per-backend rate limit, hub credential binding); otherwise the whole reachable set minus the failed backend, and a lone failed backend stays eligible. The list only narrows the API key's allowed_backends and any hub or model-experiment pin, never widens it, and never comes back empty; an x-backend preference naming a role worker is ignored on the fallback while a unified backend serves. A Dynamo shim dispatch excludes no replica, since its allow-list is a pool. The narrowed list scopes only the routed model's own attempts: chain hops (pre-stream, mid-stream, and quality) run on the caller's own list, so a chain target served only by a role worker stays reachable. The request de-duplication key now includes the effective backend allow-list when it is non-empty, so two requests with the same body and headers but different effective lists (a hub policy update without a configuration publish, experiment arms pinned to different backends, or this narrowing) no longer share a result or a cached error; unrestricted callers keep their previous key.

• Record one input decision per buffered chat request and per Anthropic retry (#1855, follow-up of #1847). A buffered /v1/chat/completions request, unified or served through disaggregated_serving, no longer runs a second input pass joined with the backend call: the pre-call gate already runs and awaits every input provider, so that pass hid no latency and only called each provider again and recorded a second decision in guardrail_verdicts_total and the audit log (a fallback_to_unified retry ran it once more). Each input provider is now called once per buffered chat request, and, as on streamed chat, the input stage no longer inspects the operator's global prompt, which is injected after the gate. guardrail::run_input_during_call and GuardrailService::run_input_during_call remain for library callers; no router path calls them. On /anthropic/v1/messages, every attempt (the primary and each fallback hop) folds its thinking pass into one request-scoped decision, also when a guardrail service became live after the main pass saw none, so a request gated on several conversion attempts records one decision, the most severe, instead of one per attempt.
• Record every request-level input decision once, under the policy that judged the request (#1847, epic #1796). A pass folded into a pending decision now records under the policy it read before its first provider call rather than the one live when it was folded, and /v1/completions and the tool and schema definition gate read theirs once at the start, so a reload during a long request (guardrails turned off for the route, or enforce switched to monitor) neither drops the decision nor labels an enforced refusal monitor. A responses_only chat request gated before an external smart-routing classifier records one decision instead of two: the bridge now runs the chat input gate and reuses or folds into the classifier pass. An /anthropic/v1/messages thinking pass gated before admission folds only when the admitted backend converts, so an attempt moved to a native backend no longer records a transform for text it sent unmasked. /anthropic/v1/messages/count_tokens no longer records a tool definition refusal (it still answers 400), so it records no decision in either mode. /v1/completions records a decision only when a check judged the request, so a request only unmatched match lists saw records nothing, as on chat. A /v1/responses/compact input the compact body cannot carry records the refusal it answers with.

• Inspect input text that still reached backends uninspected (#1836, follow-up of #1827). /anthropic/v1/messages/count_tokens is now gated like /anthropic/v1/messages on the arms that send text upstream (native Anthropic and an engine /tokenize), masking the body before it is forwarded; a refusal is a 400 Anthropic error, and the local estimate stays ungated. count_tokens records no request-level decision, since it answers no model call. On the /anthropic/v1/messages OpenAI-compatible conversion, thinking text forwarded as reasoning_content is gated and masked for each attempt on the backend it is actually admitted to (a native arm reached through a lost admission race sends the original signed text). On chat, each vLLM documents entry is read and masked in place, chat_template_kwargs string values are masked in place with their argument names inspected but never rewritten, and llama.cpp's response_format.schema is gated like response_format.json_schema. A tool's identifier-shaped top-level type and cache_control values are now read into its contract text, less a trailing _YYYYMMDD version date, so a deny term can no longer hide there. On /v1/responses, a fallback hop to a model under another policy scope (or after a policy reload) re-gates the client's tools and input under that model on every strategy, passthrough included. A request gated in several passes (a thinking block on a conversion attempt, a re-gating /v1/responses hop or continuation, or a smart-routed chat request gated before the classifier and again after routing) now records one request-level decision, the most severe, instead of one per pass; it is recorded when the request ends (after the upstream reply on a non-streaming request, when the handler returns on a streaming one).

• Inspect Anthropic document and search_result text at the input stage (#1848, follow-up of #1836). On /anthropic/v1/messages and its count_tokens route, the text blocks and title of a search_result block are read and masked in place, and its source URL is inspected but never rewritten; they used to reach native Anthropic and Bedrock backends unread. A search_result or document inside a tool_result is read on count_tokens, which forwards the raw body; /anthropic/v1/messages still refuses that placement at parse. A base64 document whose media type is text/* is decoded and inspected but never rewritten: a mask on it refuses with guardrail_mask_unappliable, and one that decodes to more than 1 MiB, would take the request's decoded documents past 4 MiB in total, names a charset other than utf-8 or us-ascii, is not valid base64, or is not NUL-free UTF-8 is uninspectable (uninspectable_input in enforce, passed in monitor). file_id references are now resolved before the input gate, once, so a text/plain, text/csv, or text/html file is read like an inline document; a guardrail refusal now happens after the files are loaded. Client-visible: document blocks with a text or content source, which were rejected at parse, are accepted: their text, title, and context are masked in place, the native arms forward title, context, citations, and cache_control unchanged (a document's block-level cache_control used to be dropped, and now also suppresses automatic cache-control injection), and the OpenAI-compatible, Gemini, and Responses conversions send the text as text parts. count_tokens now picks its arm from the backend admitted for dispatch, so admission moving a native count to an estimating backend answers the local estimate instead of posting /v1/messages/count_tokens to it. On /v1/responses, Gemini parts marked thought: true are no longer rendered as output_text, buffered or streamed.

• Inspect the remaining Anthropic text blocks and document types at the input stage (#1854, follow-up of #1848). On /anthropic/v1/messages and its count_tokens route, a base64 document whose media type is textual but not text/* (application/json, application/xml, application/javascript, application/yaml, application/toml, application/sql, application/graphql, application/x-sh, and any +json, +xml, or +yaml type) is decoded and inspected like a text/* one, and a base64 document of any other type except application/pdf is uninspectable (uninspectable_input in enforce, passed in monitor), so an unknown type fails closed; before, both reached every backend arm unread. ASCII whitespace between base64 characters is skipped for inspection, so a clean MIME-wrapped text document is no longer refused as uninspectable; the forwarded data is unchanged. Block types the router forwards verbatim to native Anthropic and Bedrock backends (server_tool_use, mcp_tool_use, mcp_tool_result, web_search_tool_result, web_fetch_tool_result, and newer ones) now have every string inspected but never rewritten, except identifiers, opaque tokens, cache_control, citations, and media_type; a nested document is read as a document. Client-visible: a search_result or document inside a tool_result, which answered 422, is accepted, read by the input stage, resolved from a file_id against the same reference and byte budgets, and rendered as text into the tool message of the OpenAI-compatible, Responses, and Gemini conversions; a nested document with a base64, url, or file_id source answers 400 invalid_request_error on those conversions instead of being dropped. A document block now keeps only source, title, context, citations, and cache_control: an unknown field, forwarded to native backends since #1848, is dropped again. AnthropicContentBlock::Document replaces its extra map with typed citations and cache_control fields, and AnthropicToolResultBlock gains SearchResult and Document variants. A raw tool-use block's input is read whole, whatever its keys. AnthropicCacheControl gains a public ttl field, so a breakpoint's ttl (for example 1h) is now kept on every block type and tool definition instead of being dropped, and a breakpoint on a block nested in a tool_result now suppresses automatic cache-control injection. The action-safety and usage.quality_judgment contexts read nested search_result and text document blocks as tool output.

• Let an oversized uncached tool set converge, and stop partial phone masks (#1850, follow-up of #1835 and #1837). When more tool-definition views are left to check than the view cap (128 at the default 4 KiB pack size), the gate now checks the first views up to the cap and remembers their clean tools before treating the request as uninspectable, instead of refusing it before checking anything; enforce still refuses that request with uninspectable_input and monitor still records one block, but a client resending the same clean tools is served after at most one refusal, with each request still bounded by the cap. Masked tools are never remembered, so a set with more masked views than the cap stays refused, and under a non-deterministic provider a packed verdict lives 60 seconds, so the resend must arrive within that window. The built-in pii phone detector now masks the country code, area code, or trunk prefix its pattern could not take (+44 20 7946 0958, +82 10-1234-5678, (02) 1234 5678, 10-1234-5678) instead of only the local part, masks each number of a run whose prefix the pattern split (02 1234 5678 02 9876 5432, 415-555-0123 212-555-0199; two numbers that together form a Luhn-valid run are still claimed by the card scanner and only partly masked), and no longer masks the tail of a dotted or longer hyphenated number (3.415-555-0123, 2024-01-1234-5678), a digit run after a version (1.0.0-rc.0123456789), a number hyphen-joined to an identifier (id-1234-5678-9012, order-1234-5678), a display resolution (3840-2160), or a year range (2019-2024); service numbers such as 1588-1234 and labelled numbers such as HP-010-1234-5678 still mask.

• Stop the built-in pii provider leaking card and phone digits through overlapping spans and card-match gaps (#1857, follow-up of #1850). A span that starts inside an earlier one and ends past it now keeps its remainder under its own type instead of being dropped, so 4111 1111 1111 1111-2222 masks the 2222 too, and an entity set to allow is set aside before overlaps are resolved, so an allowed email no longer hides the card number inside it (kim.4111111111111111@x.io). An entity set to block now blocks even when an overlapping masked span covers it, because block is decided before overlaps are resolved. The card scanner now takes the leftmost, then longest, card-shaped run of whole digit groups that passes the Luhn check (one run of 13 to 19 digits, groups of 4 with a last group of 1 to 4, 4-6-5, 4-6-4, or two or three groups of 4 digits or more such as 4111 111111111111), trying every group start so overlapping runs are masked together: a card next to a short number, after a date, or after a phone number is found (4111111111111111 010-1234-5678, 010 4111111111111111, 2026-09-29 4111 1111 1111 1111), while two phone numbers, a date and a phone number, or an ISBN are no longer one card (415-555-0123 415-555-0199, 2026-10-05 415-555-0123, ISBN 978-89-01-12345-6); a card typed with a short group (2-4-4-4, 3-4-4-4, 1-4-4-4-3, 4-4-4-3-1, 3-3-3-3-4, eight pairs of digits) is not detected. Every built-in pattern now uses ASCII word boundaries, so a card, email, SSN, AWS key, or API key written against Hangul or Han text (4111-1111-1111-1111이고, 메일kim@x.io입니다) is detected. A built-in phone span wholly inside a card span is dropped, so phone: block no longer blocks a space-grouped card and credit_card: allow no longer masks part of the card as a phone number; a phone-shaped string elsewhere still acts, so kim.01012345678@x.io under phone: block now blocks where it used to be masked as an email. A phone match that starts one group early on the preceding text masks the number after it instead (order 1234 010 1234 5678 keeps order 1234), unless the move would change how the next number is masked (a group that continues a number, or a country calling code written without its + as in 82 010 1234 5678 3, is kept and the next group taken too), and a phone number is no longer hidden by a card number written right after it (415-555-0123 378282246310005).

• Charge the token usage of a streamed /v1/responses request that an enforcing output guardrail blocks on the Chat Completions, Anthropic (including Bedrock), and Gemini strategies (#1833). The relay ended the stream with the guardrail_blocked error event and dropped its usage tap unfired, so the router recorded no per-key TPM or monthly charge, no llm_tokens_total, and no hub usage event for tokens the provider still billed. A block now charges exactly the usage the backend reported, once, the rule /v1/chat/completions and the passthrough strategy already apply; on the Gemini strategy that is the latest usageMetadata, including the one on the finishReason chunk. On the Chat Completions strategy the end-of-stream check runs on the finish chunk, before a backend following the OpenAI include_usage contract sends its trailing usage chunk, so a block raised there holds its error event until that chunk, [DONE], the post-finish grace (timeouts.request.streaming.chunk_interval clamped to 1 to 5 seconds), a read error, or the end of the stream, and charges what arrived; /v1/chat/completions charges the same stream because its check runs on [DONE]. A block raised before the backend reported any usage (a chunked window cut mid-stream) charges nothing, since nothing is estimated. A chunked block tripped by text on the finish chunk itself is a block at the terminal, so it is held and charged the trailing usage, where /v1/chat/completions stops reading and charges nothing. The client still receives a single error event.

• Charge the final usage on a blocked or completed streamed /v1/responses request instead of an earlier partial or zeroed report (#1849, follow-up of #1833). On the Chat Completions strategy, a block raised at the finish chunk used to hold for the trailing include_usage chunk only when no usage had been seen at all, so a backend that reports a partial usage object (for example prompt-only) before the finish chunk and the final one after it had its block emitted at once and charged that stale partial count instead of the trailing final one; every such block is now held regardless of what usage (if any) arrived earlier, and is charged the latest report received by then, so a trailing report supersedes an earlier partial one, never zero when a report was in fact seen. On the Gemini strategy, a completed (non-blocked) stream whose last chunk carried an empty or all-zero usageMetadata had that report wholesale-replace an earlier non-zero one, charging the completed response zero tokens; a zero report is now ignored on both the blocked and completed paths, sharing one ResponseUsage::is_zero check.

• Run the output guardrail's end-of-stream check on the terminal chunk of a streamed /v1/responses request converted to Chat Completions or Gemini (#1834). Gemini routinely puts the last text fragment on its finishReason chunk, and the relay forwarded that chunk after the check without inspecting it, so a deny-listed term or a PII value in the last fragment reached the client under every streaming_mode, masking included; a Gemini-compatible gateway that sends "finishReason": null on every delta had every delta forwarded that way. Only a string finishReason now counts as the terminal. On the Chat Completions conversion, streaming_mode: chunked completed the response while its last window was still held, dropping the tail of the reply, and monitor mode never ran its end-of-stream check, so it recorded no verdict for a completion no one had inspected. The terminal chunk now joins the check with the rest of the text, text-less terminals are forwarded after the held text, and later frames such as the trailing usage chunk still reach the client. A frame the upstream sends after [DONE] is no longer translated after response.completed. A Chat Completions backend that sends its finish chunk and then neither the usage chunk nor [DONE] no longer holds response.completed for the whole streaming total (600s by default): the router completes the reply with the usage reported so far once timeouts.request.streaming.chunk_interval, clamped to 1 to 5 seconds, has passed since the finish chunk; keepalive comments do not extend that wait.

• Inspect tool and schema definitions on every ingress (#1827, follow-up of #1805 and #1823). The input stage read only messages, so a deny-listed term, an injection payload, or PII in a tool description, a parameter schema, a legacy functions entry, or a response schema reached the model uninspected under enforce mode. Each tool and schema is now gated alone, before the messages, on /v1/chat/completions (streamed and buffered), the responses_only bridge, /v1/responses, and /anthropic/v1/messages: description and title strings and the strings inside examples are masked in place, while names, property names, required, enum-like values, and every other string are inspected but refuse the request on a mask, since rewriting them would change the tool contract. A tool every check allowed is remembered under the live policy generation (up to 65,536 per router), so a tool list resent every turn costs no provider call; a change to the guardrails configuration forgets it (a hot reload that changes only other sections no longer re-applies the guardrail policy), and blocks, masks, monitor-mode would-blocks, and fail-open passes are never remembered. More than 256 tools, or 16,384 strings over all of them, is uninspectable. The tool checks share a deadline of one provider timeout per round of eight plus three, checked only before a round is dispatched (a dispatched round always completes, so a found mask is never dropped); when every provider fails open on a whole round, the remaining tools skip the providers and see only the match lists, and a deadline refuses (guardrail_timeout) only when a fail-closed provider is selected, so an outage behind on_error: fail_open never refuses a large tool set. Tool checks count per provider; the request-level verdict is still recorded once, by the message pass.

• Bound the provider calls the tool-definition gate spends per request (#1835, follow-up of #1827). Each tool that missed the verdict cache cost one call per input provider, so a keyed client sending 256 unique tools per request forced 256 paid provider calls per provider on every request, plus up to 64 remote re-checks for each masked tool. Tools whose text totals at most 4 KiB (counted as JSON-escaped text, less when a selected provider truncates its view: System One's max_state_bytes minus 512 bytes) are now packed, up to 32 per view, into one check with every tool's contract fields at its head; a 256-tool name-only agent set costs 8 calls instead of 256, a clean pack is remembered per tool, and a mask is written back to each tool's own field (a mask on any contract field still refuses). The match lists run on each tool alone before packing, so an allow rule exempts only the tool it matches and an anchored rule reads each tool as if sent alone. A request whose tools still need more views than the view cap after the cache, the match lists, and packing is uninspectable (enforce refuses it, monitor records it); the cap is 128 at the default pack size and grows as a small provider view shrinks the pack (up to 256, one view per tool, at 2 KiB or less), so a System One target sized for a small context never refuses a tool set the 256-tool limit allows. All of a request's views share one budget of 64 re-checked strings (lifted when every masking provider re-checks locally, as for one document), so a request costs at most 128 first-pass calls per provider at the default pack size (256 under a small view) and, with T masking providers, at most 64 x T x T remote re-checks plus 64 x T verification calls. A clean tool verdict from a non-deterministic provider (anything but the built-in pii scanners without an external recognizer, and the match lists) now expires after 10 minutes instead of living until the next policy change, and after 60 seconds when it was judged in a pack of several tools, since a score-based classifier can pass a borderline tool on the pack as a whole. The tool verdicts are folded into the request's one decision on every ingress, so monitor mode records block for a would-block tool or an uninspectable tool list and enforce mode records transform when only a tool was masked; before, guardrail_verdicts_total counted neither. Custom Guardrail implementations gain two defaulted hints, max_view_bytes() and deterministic().

• Run the input-stage guardrails on streamed /v1/chat/completions requests (#1646). A stream: true request took the streaming split before the input gate, so input providers, deny and allow lists, and input PII masking never saw it and the prompt reached the backend as sent, while the same prompt without stream was blocked or masked. The pre-call gate now runs once for both arms, before the streaming split, the global prompt, every response-cache lookup, and disaggregated dispatch. A blocked streamed request gets the non-streamed block response (200 with the chat.completion block, or 400 with the content_filter envelope under block_behavior: error, plus the x-guardrail-* headers) before any SSE starts, and a transform rewrites the prompt the backend and the cache keys see. Non-streamed requests change too: an exact or prompt-prefix cache hit no longer bypasses an input block, and pays the input check like the /anthropic/v1/messages path already did. Streamed requests run no during-call stage, since a late verdict has no in-flight response left to discard.

• Charge the token usage of a streamed /v1/chat/completions request that the router bridges to an upstream /v1/responses (#1809). The bridge serves every responses_only model, every Codex/ChatGPT-subscription backend, and any request combining tools with a non-none reasoning_effort on a model flagged chat_completions_tools_require_none_reasoning, and its streamed arm recorded only a zero-token connect-time row: no control-plane usage event reached the hub, cost centers, or model budgets, and a hub key's input_tpm, output_tpm, and monthly token limits were never charged, so a client could spend past them by adding a dummy tool and stream: true. The bridge now charges the upstream's terminal response.completed (or response.incomplete) usage once at stream end through the same stream-end tap and context builder as the regular chat stream, including the served backend's backend_id, the matched hub key_id, the model-experiment arm, and TTFT; per-key stats count the request once with its tokens, and a stream that ends before any terminal usage records none. Both bridge arms now also report the upstream's input_tokens_details.cached_tokens as prompt_tokens_details.cached_tokens, on the non-streaming body and the streaming include_usage chunk, so cache reads are metered and billed as cache reads rather than as fresh input. The details object appears whenever the upstream sends input_tokens_details, which OpenAI always does, so an uncached response now carries prompt_tokens_details: {"cached_tokens": 0}, matching the native Chat Completions shape.

• Charge a slow-reading streaming client for the whole reply on every streaming relay (#1815). Ten relays read the upstream only as fast as the client read the response: the /v1/responses passthrough and its Anthropic, Gemini, and Chat Completions conversions, the /anthropic/v1/messages native passthrough (HTTP, Unix socket, Bedrock mantle), Bedrock runtime, OpenAI-wire, and responses_only arms, and the native Gemini and Bedrock runtime/Converse /v1/chat/completions streams. A client that read slowly and disconnected just before the end left the relay behind the upstream, so the terminal usage event was never observed and nothing was charged (no hub usage event, cost-center or model-budget charge, input_tpm/output_tpm/monthly charge, or per-key stats) although the upstream generated and billed the whole reply. Every relay now sends through one shared unbounded relay channel, like the regular chat stream, so it reads the upstream at the upstream's pace and charges the terminal usage whatever the client's read pace; the native Gemini and Bedrock chat pipeline now runs its tap chain in its own task instead of inside the response body, and that task also watches for the disconnect while it waits on a silent upstream, so a client that leaves during a long reasoning pause still cancels the generation at once instead of waiting for the next upstream event. A client disconnect still stops the relay at its next send and drops the upstream, so cancellation is unchanged. Per-stream buffering is now bounded by the upstream reply (its output-token cap and timeouts.streaming.total) instead of 256 events, the same bound the regular chat stream has, and the slow-client backpressure warning on the /v1/responses conversion arms is gone. A source audit (tests/streaming_relay_pacing_audit_test.rs) now fails the build on a bounded channel of SSE events or an Sse body that does not serve the shared relay channel.

• Charge the token usage of a streamed /v1/responses request that the router converts to Chat Completions (#1825). That conversion serves vLLM, SGLang, TensorRT-LLM, Ollama, llama.cpp, mlxcel, LM Studio, continuumrouter, and generic backends, and every such request was charged zero tokens: the router never sent stream_options, so an OpenAI-contract server reported no usage, and the relay built response.completed from the finish chunk before reading that chunk's usage and stopped reading there, so a trailing include_usage chunk never arrived. The result was one hub usage event with zero prompt and completion tokens, zero charged to the key's token windows, monthly budget, cost center, and model budget, and zero usage in the client's response.completed. The relay now sends stream_options.include_usage: true (as /v1/chat/completions already does for every OpenAI-compatible stream) and completes the response on the first chunk at or after the finish chunk that carries a usage object, or on [DONE], so a trailing usage chunk, usage on the finish chunk, and usage on every chunk are each charged exactly once. "usage": null deltas and all-zero or empty usage placeholders no longer reset the counts or end the stream early, and a stream with no usage still completes on [DONE] or at end of stream with zero usage, as before. The relay now reads past the finish chunk; an upstream read error while it waits still completes the response with the usage absorbed so far.

• Run the guardrail stages on /v1/responses and /v1/completions (#1805). Neither route ran the input stage, /v1/completions ran no output stage, and a non-streaming /v1/responses reply was never gated, so a client (Codex and every other Responses-API client included) could skip input and output blocking and masking, under local policy and under an enforce-mode hub guardrail_policy_v1 alike, by choosing the endpoint, with no metric, audit row, or hub counter recording the bypass. /v1/responses now runs the input stage in both modes before the response cache, backend selection, and every fallback attempt, and the output stage on all four non-streaming strategies before the store: true session write, so GET /v1/responses/{id} and a previous_response_id continuation carry the gated text. /v1/completions gates its prompt before dispatch and its reply after, including a stream: true reply the funnel buffers. Each route renders a block in its own wire: a response with status: incomplete and incomplete_details.reason: content_filter, or a text_completion with finish_reason: content_filter, and the OpenAI content_filter error envelope (HTTP 400) under block_behavior: error. A block is never cached and never triggers a fallback_chains hop, even when 400 is listed in trigger_conditions.error_codes. Every Responses input item is inspected from its wire form, so a raw item the router forwards verbatim (a function_call_output with status: incomplete, or any item carrying phase, agent, or caller) no longer passes unread, and tool-call arguments (Responses items and chat tool_calls) are read token by token from the raw string, keys, numbers, and duplicate keys included, and masked in place without re-encoding; an arguments string the JSON parser rejects is refused in enforce mode. An unknown item type is refused by the parser with 422. On the conversion strategies a previous_response_id continuation is re-checked over the rebuilt history under the dispatched model's policy. An input mask is applied where each string lives (instructions, every turn, tool outputs, each token of tool-call arguments) on /v1/chat/completions, the responses_only bridge, /v1/decisions, /v1/responses (stored history included), compact, and /v1/completions, and a mask that cannot be applied that way is refused rather than dispatched unmasked. Only the providers that returned the mask are re-run, within a deadline of three provider timeouts, a provider failure after a mask refuses even under fail-open, and the providers see one message per client message. Chat also reads reasoning_content, refusal, custom tool input, and tool and participant names, /v1/completions judges each prompt on its own and the suffix once (even with no prompt), reads nested prompt arrays, judges at most 128 prompts per request, and records one decision per request, and a request read as more than 16,384 strings is treated as uninspectable. /v1/completions token-id and prompt_embeds prompts are refused in enforce mode when an input-stage check applies, and /v1/responses/compact input is gated. A mask now also clears logprobs and top_logprobs on the chat, Completions, and Responses wires, streaming included. An output block on /anthropic/v1/messages and an input block on /v1/decisions no longer hop to a fallback model, and a 200 block no longer carries x-served-backend. Monitor mode is unchanged. The guardrail, architecture, and API docs now tabulate the real per-route, per-stage coverage, including the routes that stay ungated (/v1/batches).

• Fold the context-compression history summary into the request's leading system (or developer) message instead of inserting a second system entry after it (#1704). A hub compression policy with summarization turned a multi-turn [system, user, assistant, user] request into [system, system, ...], which a strict chat template such as Qwen3.8's rejects with System message must be at the beginning.. The merge is unconditional, reuses the requires_single_leading_system merge (blank-line boundary, parts arrays kept), and leaves a request with no leading system message unchanged: its summary is still a synthetic system entry at position 0. The per-backend requires_single_leading_system collapse already runs on the compressed payload at every send site, so a mid-conversation system entry that compression keeps is merged there as before.

• Count a Gemini thinking model's thoughtsTokenCount in chat-completions usage (#1677). Native Gemini usage on /v1/chat/completions (buffered and streaming) and on the /anthropic/v1/messages bridge now reports completion_tokens (and the bridge's output_tokens) as candidatesTokenCount + thoughtsTokenCount, takes Gemini's totalTokenCount as the total when it is non-zero, and adds completion_tokens_details.reasoning_tokens, matching what /v1/responses already reported. This is billing-visible: a thinking model's metered output on the chat route and the bridge, locally and in the Hub usage record, rises by its thinking tokens.

• Answer a safety-blocked Gemini prompt with a content_filter finish instead of a raw Gemini object (#1677). A native response or stream event that carries promptFeedback.blockReason and no candidate now becomes an ordinary chat.completion (or a finish chunk ahead of the usage chunk when streaming) with finish_reason: "content_filter" for every block reason, and its usage is still metered; before, the non-streaming body and a stream event without usageMetadata were returned verbatim and a stream event with usage produced no finish reason.

• Resolve requires_single_leading_system from the compiled-in catalog on every OpenAI-wire backend type, not only openai (#1705). The key now resolves field-wise through all three sources (the backend's model_configs, the model-metadata.yaml cache, then the compiled-in catalog), and the last step is consulted for vllm, sglang, generic, and the other OpenAI-wire types; the wholesale compiled-in lookup stays openai-only so a self-hosted server that names a model after an OpenAI one never inherits its responses_only dispatch. The Qwen3.8 default itself lives in the shipped model-metadata.yaml, which already resolved on every backend type but is read only when model_metadata_file names it; the backend docs and config.yaml.example now say so, since a router started without the file forwards a strict-template model's system entries as sent.

• Report usage on a streamed /v1/chat/completions response that an enforcing output guardrail blocked (#1680). The block replaced everything the gate held with the content_filter terminal, and the usage report went with it, so the client saw no usage for tokens the backend generated and billed. The gate now remembers the latest usage the source reported and, when the block dropped the payload that carried it, appends an OpenAI usage-only chunk (choices: []) after the terminal and before [DONE], under every block_behavior; a usage report that already reached the client is not repeated. On the native Anthropic source path the router also meters the blocked completion from those counts, where it previously recorded nothing because the block dropped the message_stop its usage is built from. A chunked block raised mid-stream still carries no usage: the stream is cut before the backend reports it.

• Stream a typed native backend on a unix:// socket incrementally, under the router's streaming timeouts (#1679). The typed streaming path (execute_streaming_request on Gemini, Bedrock mantle, Anthropic, OpenAI, vLLM, llama.cpp, and Continuum Router backends) used the executor's buffered Unix-socket request, which read the whole answer under a hardcoded 30s timeout before the first byte left the router: SSE arrived as one burst at the end, timeouts.streaming.first_byte measured nothing, and any answer longer than 30s failed. RequestExecutor gains execute_streaming / execute_streaming_with_client, which return an ExecutorStreamResponse once the response headers arrive and yield the body as the engine writes it, plus with_unix_socket_config, which each typed backend sets to its HTTP streaming client's window (300s for Gemini, 600s for Anthropic and Bedrock mantle) so the socket is never stricter than HTTP. The HTTP path and the buffered non-streaming socket path are unchanged; a non-2xx streaming answer on the Anthropic and Bedrock mantle paths now reads the same bounded 64 KiB error-body prefix Gemini already did.

• Send the Dynamo direct-mode instance-id headers as the decimal lease id the frontend parses (#1748). x-dynamo-prefill-instance-id and x-dynamo-worker-instance-id carried the registry's lowercase-hex id, which dynamo.frontend reads as absent, so every shim request on a real Dynamo 1.5 fleet ended in the direct-mode refusal. The Admin view and the x-continuum-* response headers keep the hex spelling.

• Keep a Dynamo 1.5 prefill worker's model card instead of discarding its typed fields (#1748). Dynamo 1.5 writes runtime_config.disaggregated_endpoint on a prefill card as an object holding the bootstrap dial address, and the strict string field reset the whole typed card, so the prefill worker lost its model name and capacity and the shim picker never chose it. Any non-string value is now read as absent, and the address is not carried toward the Admin view.

• Make the deploy/examples/dynamo-direct-shim/ workers start (#1748). The tokenizer cache volume was mounted at a path the image lacks, so Docker created it root-owned and the non-root mocker exited with Permission denied; it now mounts over /home/dynamo/.cache, which the image owns. The example, the harness, and dynamo-e2e.yml no longer require an NGC login, since the pinned dynamo-planner:1.5.0 tag pulls anonymously.

• Meter streaming /anthropic/v1/messages bridged from an OpenAI-compatible or OpenAI Responses backend the same as the non-streaming request when the upstream reports a cache read above the prompt (#1762). The stream transformers kept only the Anthropic exclusive split, which saturates on such a report, and re-summed it, so an upstream prompt_tokens: 100 with cached_tokens: 800 was metered as an 800-token prompt with an 800-token cache read, a total of 810, and 800 cached input on the Hub, where the non-streaming path records 100, 100, 110, and 100. The transformers now keep the upstream's inclusive prompt beside the split, and both stream-end snapshots are built by one shared helper that hands that count to the existing cache-subset clamp in TokenUsage::from_json_value. The usage the client sees on the Anthropic wire is unchanged.

• Register the disaggregated RoleFilterScorer as phase-only (#1759). On a deployment with disaggregated serving and prefix_routing but no KV index it was the only registered scorer, so every keyed request ran the scorer pipeline, scored every candidate a neutral zero, and was recorded as a fallback routing decision with scorer_fell_back set; a pass without an inference phase now leaves phase-only scorers out and reports the pipeline as not attempted. Routing is unchanged.

• Classify the Dynamo direct-mode refusal on the streaming disaggregated arm (#1759). The shim's refusal arrives as a 200 SSE stream carrying the upstream error, which the streaming arm treated as a served stream; the streaming pipeline now marks a relayed upstream failure, and the arm reads the head of a failed or marked shim response and answers the refusal as 503 disaggregated_prefill_decode_unavailable with disaggregated_errors_total{phase="orchestration"}, as the buffered arm does. Any other response is replayed to the client unchanged.

• Stop charging the upstream twice on a buffered disaggregated backend mismatch (#1759). With fallback_to_unified on, a response served by a backend the selection did not permit is now returned as a fallback answer instead of being discarded and re-issued; without it the request is still refused with disaggregated_backend_mismatch.

• Refuse a post-handshake command flood on the Dynamo load subscriber (#1759). More than 256 command frames within one second after the ZMTP handshake is now a transport_refused refusal, so a peer can no longer hold a subscription in its skip loop forever while the gauge reads connected; libzmq heartbeats stay far below the bound. MAX_COMMAND_BYTES, GREETING_BYTES, ZMTP_VERSION, and the new MAX_COMMANDS_PER_WINDOW and COMMAND_WINDOW are re-exported from dynamo::load.

• Apply a configuration reload published while the router starts, between the boot capture and the Dynamo discovery supervisor's subscription (#1759). It used to wait for the next change; the supervisor now compares the channel's current value with the boot section on start.

• Stop serving guardrail-bypassing replies from the buffered disaggregated path and the response cache (#1817). A non-streaming /v1/chat/completions reply served through disaggregated_serving (fast decode, prefill-then-decode, a Dynamo shim dispatch, or a backend-mismatch reply served under fallback_to_unified) skipped the during-call and output stages; it now runs both, exactly like a unified reply. A response-cache hit (exact, prompt-prefix, or semantic, on buffered and streamed chat, /anthropic/v1/messages, and /v1/responses) skips the output stage, and the cache key did not reflect guardrail policy, so an entry stored under mode: monitor, under a since-tightened policy, or by a bypass_api_keys caller in a shared namespace kept being served until its TTL. Every cache namespace now folds in a guardrail policy scope: a digest of the live composed policy (hub guardrail_policy_v1 statements and governed patterns included, the bypass key list excluded), the matched routes entry, and whether the caller holds a bypass key. The digest also covers the router version, so a release that changes the built-in detectors never serves an older binary's entries, including from a Redis cache a mixed-version fleet shares. A reply is not stored when the guardrail policy changed while it was in flight (a key added to bypass_api_keys, or a switch to monitor, mid-request), or when an enforced output check passed it only because a provider failed under on_error: fail_open; while a provider is failing, the cache stops filling router-wide. A change outside guardrails, such as a new classifier model behind the same endpoint or an edit to the backends[] entry behind a provider, does not move the scope. Guardrail-enabled deployments lose their existing cache entries on every upgrade and after each policy change (one miss per distinct prompt); deployments without guardrails keep their cache keys byte for byte. When a failed buffered disaggregated dispatch falls back to unified serving, the during-call stage runs a second time on the unified attempt (one extra evaluation and duplicate audit rows). The public ResponseCacheKey struct has a new store_permit field (StorePermit::default() for an unconditional store), so code that builds it with a struct literal must set it. The guardrails guide no longer claims the output stage runs before a cached copy is returned.

• Run the guardrail input stage before a smart-routing classifier that sends the prompt out of the router (#1816). On /v1/chat/completions the LLM classifier, the System One classifier, and a shadow: true System One classifier received the raw prompt before any deny rule, input provider, or PII mask applied, including a prompt the gate then blocked. When the classifier can leave the process, the input stage now runs first, keyed on smart_routing.virtual_model rather than the name the client sent, so under intercept_all a client cannot pick a looser guardrails.routes entry for it: a block stops the request before the classifier is called, and a mask is applied by the per-message masking core (#1805) to every message where it lives, system included, before the classifier reads it, or refuses the request when it cannot be applied that way. After routing the stage runs again only when the routed model resolves to a different guardrails.routes entry, so a request is evaluated once unless a per-route override applies, and then it is counted twice in the guardrail metrics. A method: rule router without a shadow is gated once after routing, as before.

• Mask every segment of an /anthropic/v1/messages request in place (#1823, follow-up of #1805). A masking verdict (the built-in pii provider, AWS Bedrock Guardrails) replaced the last user message with the redacted form of the whole conversation, so PII in system, an earlier turn, or a tool_result still reached the backend unmasked while the verdict and metrics recorded a mask, the last user turn received a copy of every earlier turn, and its tool_result blocks collapsed into one string. The route now uses the per-leaf masking core: system, every turn's text (including a mid-conversation system entry and a bare string inside a content array), tool_result content, and the string values of tool_use.input are read and each is masked where it lives, on every backend arm, streamed or not. The keys, numbers, and booleans of tool_use.input are never read. Thinking text is not inspected (its signature is verified by the native backends), including where the OpenAI-compatible conversion forwards it as reasoning_content; redacted_thinking, images, documents, and block types forwarded verbatim are never read or rewritten. A mask that cannot be applied (one spanning two strings, for example) is answered with the Anthropic block and nothing is dispatched. Block-only providers now also inspect system, tool results, and tool-use input on this route. /anthropic/v1/messages/count_tokens is still not gated. The 64-string limit on masked strings per request no longer applies when every masking provider re-checks in process (the built-in pii provider without an external recognizer), so an agent session whose resent tool results carry many phone-shaped numbers is masked rather than refused on every turn; a provider that may call out keeps the limit. The built-in phone detector also matches IPv4 addresses and timestamps, and the guardrails guide now recommends options.actions: {phone: allow} for agent traffic.

• Keep health checks and model catalog fetches off the proxy connection pool (#1924, epic #1703). The health checker and the /v1/models fetcher sent their probes through clones of the proxy reqwest client, which share its pool, so when proxied requests filled a backend's single HTTP/2 connection up to the server's MAX_CONCURRENT_STREAMS (128 at an nginx ingress) the probes waited for a stream slot, timed out, and a busy but live backend was marked unhealthy; with its only backend unhealthy the router refused every request until the long streams ended. Probes now run on a dedicated client built by the same builder as the proxy client (same TLS roots, timeouts, and server.http_client tuning) with its own connection pool, used by the background health checker, model discovery, and the one-off candidate health probes of the admin backend probe and the control-plane activation preflight. Credentials and the per-request health_checks.timeout (#1065) are attached per request as before, and connection pre-warm still warms the proxy pool. An embedder that injects a client with with_http_client keeps probes on that client unless it also calls the new ContinuumRouterBuilder::with_probe_http_client, since its TLS or proxy settings cannot be rebuilt from the configuration.

• Report an applied Hub backend revision as active from BackendSyncStore::status when the serving pool matches it (#1936, follow-up of #1808). Since #1808 the public wrapper passed a hardcoded "pool not converged", so it could never report active even for an applied revision whose pool matched, and the four tests that asserted active.is_none() through it could not fail. The method now evaluates the live pool itself, and the inventory heartbeat calls it instead of duplicating the convergence check, so both share one implementation. Library surface, for control-plane users of continuum_router::control_plane::BackendSyncStore: status is now async and takes the live &BackendPool as a new second parameter.

• Bound the token ids continuum-kv-listener retains per source (#1936, follow-up of #1806). A single-block BlockStored event kept its whole token_ids as its own segment whatever its length, so the documented worst case was max_tracked_blocks x max_chain_tokens (about 6.1 GiB of ids per source at the defaults). A new per-source max_retained_tokens (default 16777216, 64 MiB of 4-byte ids; 0 is a load error) retires the oldest tracked blocks the same way max_tracked_blocks does, and purges only a prefix whose last tracked reference retired, so the ceiling is now min(max_retained_tokens, max_tracked_blocks x max_chain_tokens). A single-block event whose non-empty token span differs from a non-zero block_size (vLLM skips masked sliding-window blocks from the hash list while the span stays contiguous) is now tracked without tokens instead of retaining the whole span, which leaves its descendants unattributable; it is logged once per source.

• Keep text-less chunks behind held text on an enforcing chunked output guardrail (#1643, PR #1959). With streaming_stream_first: false, finish_reason and usage (or an Anthropic content_block_stop and message_delta) left before the held partial window, so a client that ends a completion on its finish chunk lost up to streaming_chunk_size - 1 trailing characters. Such chunks now queue behind held text and leave with it in order; with nothing held they still leave at once. Tool-call deltas that follow held text now arrive with the next window or at end of stream.
• Fit max_tokens budgets to each fallback target's context window instead of sending the primary's budget unchanged (#1601, PRs #1807 and #1908). Compatibility is judged on a typical prompt estimate (ASCII at three characters per token, any other character as one token), the clamp keeps a byte upper bound, a target with under 16 tokens of room is skipped as incompatible, and budgets are only ever lowered. Limits come from the configuration snapshot the request was admitted under, with backend metadata over global metadata and raw engine limits kept when configured metadata omits them. A quality-check hop skips an incompatible target within the ordinary attempt bound. The mid-stream relay no longer awaits a full model aggregation before it needs one, and its 503 for an exhausted walk no longer names a chain model the caller's key may not see.
• Warm the model catalog when the router is built, so the first fallback hop is clamped to an engine-reported window on the embedded into_router() path and with model_aggregation.background_refresh disabled (#1935, PR #1938). ContinuumRouter::build() spawns one background revalidation that never blocks the build.
• Record each quality-fallback generation's tokens under the model that generated it, and keep the tokens of an unusable generation (PR #1828, follow-up of #1601). A fallback.quality_check hop's usage was recorded under the originally requested model, and a /v1/decisions emulated reply that could not be parsed or validated dropped the usage the backend reported. Such a generation is now one failure row carrying its tokens, without a second zero-token failure row, and a skipped incompatible target records nothing.
• Restore the Hub-assigned backend pool after a restart (#1558, PR #1808). Persisted Hub backend layers are reconciled against the live pool before the hot-reload loop starts, an identical revision is retried while the pool has not converged, and the heartbeat reports a revision active only for the exact persisted revision and effective pool. A hot-reload-added generic backend whose detected type differs from generic is no longer drained and rebuilt on every reload.
• Keep continuum-kv-listener prefixes across long vLLM block chains (#1627, PR #1806). The listener copied cumulative tokens into every tracked block and refused a valid batch at max_chain_tokens, so the next sequence looked like a gap and cleared every prefix for the backend. Blocks now hold their own token segments with generation-bound ancestry, an over-limit chain becomes unattributable without losing sequence progress, capacity pressure retires the oldest prefix instead of clearing unrelated state, a descendant of a missing or replaced ancestor no longer regains salt attribution, and an empty or short token segment is refused.
• Back off when the Dynamo discovery watch keeps ending (#1754, PR #1770). Only a watch session as long as watch_reconnect_backoff.max returns the delay to its floor; a proxy that closes every watch right after the range get is no longer redialled every 500ms. A backoff-only reload keeps the registry snapshot, a disable is no longer counted as kind="restart", and a watcher whose cancel channel was dropped exits instead of spinning.
• Refuse duplicate Dynamo instance and publisher ids instead of resolving them by etcd key order (#1753, PR #1766). Two base model cards for one (component, endpoint, instance_id), two generate endpoints with one instance id, or two kv_metrics channels with one publisher id are all refused and counted once as continuum_dynamo_discovery_events_total{kind="duplicate"}. LoRA adapter cards no longer join the worker, which fixes an adapter card overwriting the worker's model name and capacity.
• Cancel a silent /v1/responses passthrough stream when the client disconnects (#2003, PR #2004). The relay awaited only the next upstream event, so a client that left while the provider was silent kept the upstream request alive until the provider emitted again; the relay now watches the downstream close beside each upstream read and drops the upstream at once. A background: true stream is not cancelled, since that response belongs to the provider after dispatch and is fetched later.

Security¶

• Close a crash window in which the opt-in key-table sidecar could keep a key the Hub revoked (#1969). With control_plane.policy.persist_key_table: true, a security-relevant key-table change (a key revoked, removed, re-hashed, or re-tiered, or a tier limit changed) reached the sidecar only through a blocking-pool write that nothing awaited, after the change was already enforced and its cursor acknowledged to the Hub, so a process killed in between (or a SIGTERM that cancelled the queued write) restarted during a Hub outage into the previous table and re-admitted the key. Such a change now removes the previous file and fsyncs its directory on the applying thread before the new snapshot is swapped in and before the acknowledgment is published, and advances the writer's generation watermark so an older table still queued cannot recreate it; the new table is then written on the blocking pool as before. A crash in the window leaves no file, so the restart is a cold start. A removal failure is counted in policy_sync.key_table_persistence.write_failures and warned once per failing streak, and never delays enforcement. A routine refresh still never removes the file. The restored table is also bounded against clock changes: a sidecar whose sync time is more than 5 seconds ahead of the restoring clock is refused as unusable instead of clamped, and a restored table also expires at a monotonic deadline taken at restore, so a wall clock stepped backwards cannot lengthen how long it serves.

• Encode a model id as exactly one path segment in every upstream URL built from it (#1878). The Nano Banana image routes (/v1/images/generations, /v1/images/variations, /v1/images/edits) formatted the client's model into the Google Gemini URL unencoded, so an id such as gemini-x-image/../../cachedContents?a= passed the gemini-*image* gate and the router sent the request, with the operator's x-goog-api-key, to /v1beta/cachedContents instead of generateContent. Every such builder now goes through one helper, encode_model_path_segment (/ becomes %2F, ? %3F, # %23, % %25): the image routes, the native Gemini Responses and multimodal embedding endpoints, and each backend's get_model_details, which also refuses an empty, ., or .. id before sending anything. Ordinary ids produce byte-identical URLs.

• Select the backend for Nano Banana image requests by its configured type, never by its name or URL (#1896). The Nano Banana routes (/v1/images/generations, /v1/images/edits, /v1/images/variations with a Nano Banana model) send every request to Google's Gemini API carrying the selected backend's key as x-goog-api-key, and they picked that backend when its name contained gemini or its URL contained generativelanguage.googleapis.com anywhere, so an openai, generic, or vllm backend named gemini-gateway, or one whose URL only embedded that string (https://gw.example.com/generativelanguage.googleapis.com/), had its own key sent to Google. Only a type: gemini backend is now a candidate, and the same rule no longer keeps a type: openai backend whose name contains gemini out of dall-e-2 variations. Migration: a type: openai or generic backend pointed at Google's OpenAI-compatible endpoint (https://generativelanguage.googleapis.com/v1beta/openai) stops serving Nano Banana until it is changed to type: gemini. A type: openai backend at that endpoint is also no longer excluded from OpenAI dall-e-2 variations, so in a pool that mixes it with OpenAI backends some variations requests may be routed to it and fail until it is retyped.

• Refuse . and .. as a Bedrock model id (#1896). The Bedrock runtime and converse paths refused only an empty id, and . or .. percent-encodes to itself, so a configured id of that form resolved as a dot segment in /model/{id}/invoke or /model/{id}/converse and addressed another path. Both are now rejected with a validation error on model before any request is signed or sent; every other id encodes as before.

• Stop a client from minting smart_routing_requests_total series with arbitrary model names (#1897). With smart_routing.intercept_all every request is smart-routed, and its source_model label carried the client's raw model with no cardinality bound, so an authenticated client could grow router memory and every scrape by one never-evicted series per distinct name. The label now keeps a requested model verbatim only when it is smart_routing.virtual_model or a model the router currently serves; a request without model exports none, and any other name, a model alias included, exports other. A served model literally named other or none shares that series. The x-smart-route-source debug header and the routing log lines keep the name the client sent.

• Strip client-supplied x-dynamo-* headers on every request in every build (#1750, PR #1765). They were stripped only while dynamo.shim.backends was configured, so an ordinary backend pointed at a Dynamo direct-mode frontend received the client's x-dynamo-prefill-instance-id and x-dynamo-worker-instance-id and let the client choose the worker pair.
• Send the backend-naming response headers of disaggregated serving only when the client sent an x-backend request header (#1751, PR #1767). Client-visible: x-continuum-prefill-backend, x-continuum-decode-backend, and x-served-backend (which on a Dynamo shim dispatch carry worker instance ids) now follow the rule x-served-backend already followed elsewhere; a blank x-backend opts in. x-continuum-routing-path stays unconditional.

Dependencies¶

• Refresh every dependency to its newest release, including dirs 7, opentelemetry 0.33, and tracing-opentelemetry 0.34, plus the WebUI's Alpine.js 3.17.4, uPlot 1.6.32, and Tailwind CLI 4.3.3 (PR #1771). OTLP 0.33 retries transient collector failures inside the exporter timeout, so tracing.otlp.timeout now bounds one batch export including its retries. The regenerated stylesheet restores about 20 classes the committed tailwind.css was missing.

v1.31.0 - 2026-09-25¶

Added¶

• Add make calibrate-systemone and a recorded calibration of the three System One consumers (#1725, epic #1719). The run replays the labelled corpora exactly as each consumer sends them (its own state builder and question battery) against a /v1/systemone endpoint, records accuracy, the constant-answer baseline, Brier, ECE, AUROC, confusion matrices, p50/p95 latency and tokens per call per slice (all, en, ko, non_latin) with the checkpoint revision, and derives thresholds with tested rules. Runs against stock laya-serve (CPU), each pinned Laya checkpoint, and Kev-0.8B are committed under tests/fixtures/systemone/calibration/2026-09-24/; hosted Jev, Kev-4B and the method: llm shadow arm were not measured and say why. The new operator page docs/en/decision-models.md covers the models, the wire, the three consumers, egress and adversarial caveats, the procedure, the results, and when to use a decision model.

• Add the systemone guardrail provider (aliases jev, laya; #1722, epic #1719): one call to a System One decision model (hosted Jev, or a self-hosted Laya / Kev / Nimble server through the shared client from #1720) asks one noul per router category plus a four-level severity score, and the answered probability is the verdict score. p >= category_thresholds[category] (default 0.70) blocks, p >= options.review_threshold (default 0.35, per-category options.review_thresholds) flags, and a flag with severity >= options.severity_block_level (default 2.0) is upgraded to a block, which makes the Flag verdict and a real review band reachable from a model for the first time (the classifier providers emit a constant 1.0). The target is the shared systemone block on the provider row. The state is an object, never joined text ({user_message, history} at input; {ai_response, user_message} at output and, through check_streaming_chunk, per chunked window), excludes system prompts and tool results unless opted in, and is fitted under max_state_bytes before dispatch. options.task: injection narrows the battery to jailbreak plus indirect_injection; options.questions replaces wording verbatim. Every 422 / 429 / 529 / timeout / malformed answer is a check error under on_error, and a 529 also records guardrail_degraded_total{kind="overloaded"}. config validate refuses a missing target block, row-level target fields, a review threshold above the same category's action threshold, and an unknown question category. Ships in mode: monitor with the new guardrails-systemone template, since the models are a first-pass gate rather than adversarially robust.

• Add the System One decision-model classifier for smart routing (smart_routing.classifier.method: system_one and hybrid_system_one, #1723, epic #1719). One decision call per uncached request asks a score over the ordered complexity levels (the five built-ins plus custom_complexity_levels in rank order, at most ten) and a choice over the domains (the six built-ins plus custom_domains, a load-time warning past sixteen), rounds the score to the nearest level, and fills the same ClassificationResult the other classifiers produce with a real confidence (the minimum of the two answers'), so routing policies need no change; two optional noul questions (signals.needs_tools, signals.is_sensitive) are recorded as classification signals only. min_confidence (default 0.5) gates the answer: system_one falls back to the rule result below it, hybrid_system_one (rule first, System One when the rule is unsure) escalates to the LLM classifier when one is configured and otherwise uses the rule result, and any error returns the rule result exactly as a failed LLM classification does. The state sent is the LLM classifier's egress as a JSON object (the last user request truncated by max_input_tokens, the 200-character system excerpt, message_count, has_images, has_tools, the requested model) and nothing else; the cache key is its SHA-256. shadow: true beside rule, llm, or hybrid runs the battery without changing any routing decision and reports agreement in smart_routing_shadow_agreement_total{field, agree} plus the X-Smart-Route-Shadow debug header, which is how a fleet measures the decision model before switching to it. The block embeds the shared systemone target contract from #1720 (flattened, so classifier.system_one.endpoint or .backend), mirrors the LLM classifier's cache and disable_under_load_state fields, hot-reloads with the rest of smart_routing (a reload from method: llm to system_one and back rebuilds exactly the classifiers the new method needs), and validates at load, which also closes a gap for the LLM classifier: classifier.llm.backend naming an unknown or enabled: false backend is now a load error under a method that dispatches to it, instead of a silent fallback to rule-based classification. New metrics: smart_routing_system_one_classifier_cache_hits_total, smart_routing_system_one_classifier_fallbacks_total{reason}, smart_routing_system_one_classifier_input_tokens_total{counted} (server-reported usage recorded as router-internal, counted following count_toward_budgets), and classifier_type="system_one" on the existing classification series. A smart-routing-systemone configuration template ships with the configuration assistant, and tests/smart_routing_accuracy_test.rs gains a System One mode replayed through the fixture server over the same labelled cases.

• Add a System One evaluator for safeguards.action_safety (#1724, epic #1719). evaluator: systemone with a systemone block (the shared System One target plus allow_threshold 0.85, deny_threshold 0.60, min_confidence 0.6, and mode: advisory | enforce) asks one decision call per proposed tool action: action as approve / confirm / block, plus irreversible and injected_instruction, over {tool, arguments, user_request, recent_tool_results} with no system prompt. The mapping is fail-closed: allow only for an approve choice that clears every threshold with no injected instruction, deny on a block probability or an injected instruction at deny_threshold, and indeterminate for confirm and every transport or parse failure. user_request is the last user message that is not a tool round trip (tail-capped at 4 KiB) and Claude Code <system-reminder> blocks are skipped. The default advisory mode emits the chat-model evaluator's verdict (or indeterminate without a backend) and records the would-be verdict in the diagnostics and the new safeguards_action_safety_advisory_total{would_verdict, actual_verdict} metric; enforce emits the mapped verdict. evaluator: llm stays the default, backend becomes optional with systemone, and every existing bound applies unchanged.

• Add the Dynamo direct-mode frontend shim pool and instance-id header picker (#1699, part of epic #1703). With disaggregated_serving.enabled: true and a dynamo.shim.backends pool of ordinary OpenAI-wire role: unified backends running dynamo.frontend --router-mode direct, the router picks a prefill worker (active_prefill_tokens / max_num_batched_tokens) and a decode worker ((active_decode_blocks + ceil(estimated_prompt_tokens / block_size)) / total_kv_blocks, one candidate per data-parallel rank) from the discovery registry, preferring the lowest fresh cost and rotating round-robin among candidates with no fresh sample, and sends the client request unchanged through the pool with x-dynamo-prefill-instance-id, x-dynamo-prefill-dp-rank, x-dynamo-worker-instance-id, and x-dynamo-dp-rank set after stripping any client-supplied x-dynamo-* header. That strip is mounted on the whole /v1 surface whenever a pool is configured, so no route carries a client pin to a shim, not even one that selects a shim replica as an ordinary backend. dynamo.shim.model_map decides candidate matching only; the request body is forwarded unchanged, so the frontend must be able to resolve the client's own model id. A model no worker serves falls through to ordinary routing; with no prefill worker only x-dynamo-worker-instance-id is sent and the path reports unified. Responses carry x-continuum-routing-path, x-continuum-prefill-backend / x-continuum-decode-backend as dynamo:<instance_id>[/dp<rank>], and x-served-backend naming the shim. The frontend's direct-mode refusal is answered as 503 disaggregated_prefill_decode_unavailable and counted as disaggregated_errors_total{phase="orchestration"} without re-pairing. New continuum_dynamo_selection_total{role, reason}; selection is pluggable through the DynamoWorkerPicker trait on the registry. dynamo.shim reloads immediately, dynamo.shim.model_map translates client model ids to Dynamo display names, and a new dynamo-direct-shim.yaml template is included.

• Add a GPU-free end-to-end harness, deployment example, and operator documentation for Dynamo prefill/decode fleets behind a direct-mode frontend shim (#1700, part of epic #1703). deploy/examples/dynamo-direct-shim/ brings up etcd, two dynamo.mocker workers in prefill and decode disaggregation modes, one dynamo.frontend --router-mode direct, and the router, with every image tag pinned in a single .env so the compose stack and the new compatibility table cannot drift; --event-plane zmq is passed explicitly because the mocker defaults to NATS whenever discovery is etcd while the router's load subscriber reads ZMQ only. scripts/e2e-dynamo-shim.sh waits for GET /admin/dynamo/workers to show both roles, sends a streaming and a non-streaming completion, asserts that x-continuum-prefill-backend and x-continuum-decode-backend name two distinct instances and that no x-dynamo-* header is echoed to the client, then stops the prefill worker and asserts the degradation to aggregated selection, tearing the stack down on exit including on failure. .github/workflows/dynamo-e2e.yml runs it on manual dispatch only, with the runner labels as an input because the repository's self-hosted pool is macOS while the Dynamo images are linux/amd64. docs/en/architecture/dynamo-integration.md gains a topology diagram, a compatibility table, and an explicit list of what stays Dynamo-owned (tokenization, the prefill/decode handoff, KV transfer, worker liveness); docs/en/deployment.md gains a "Dynamo prefill/decode fleets" section; Korean mirrors updated. Kubernetes equivalents ship as deploy/helm/continuum-router/values-dynamo-direct-shim.yaml (with an opt-in dynamoShimService ExternalName alias, off by default) and deploy/kubernetes/overlays/dynamo-direct-shim/, both pointing at an operator-managed etcd and an existing shim Service with no RBAC change.

• Add a bounded asynchronous process-log pipeline and an optional rotating file sink (#1676). Formatted log records now enter a bounded queue that one dedicated writer thread drains, so a slow or stalled stdout, stderr, or file destination no longer blocks a Tokio worker serving requests. The queue is bounded by record count (logging.queue_capacity, default 8192) and by retained bytes (logging.queue_capacity_bytes, default 16 MiB), each record is capped at 64 KiB and cut on a UTF-8 boundary, and a full queue or a failing sink drops records instead of blocking or stopping the router. logging.file writes plain-text records to a directory with size-based rotation, bounded retention, {date} / {time} / {pid} / {sequence} filename placeholders ({sequence} is required), and file mode 0600. Both queue settings require a restart. New metrics: continuum_process_log_queue_depth, continuum_process_log_queue_bytes, continuum_process_log_dropped_total, continuum_process_log_truncated_total, and continuum_process_log_write_errors_total.

Changed¶

• Ship measured System One defaults (#1725). Against the reference run (the compose file's laya-serve on CPU), the guardrail review_threshold default moves from 0.35 to 0.40, the violence action threshold from 0.70 to 0.75 (the other categories keep the vendor 0.70, since none reached 0.90 block precision), and smart_routing.classifier.system_one.min_confidence from 0.5 to 0.15, where the System One chain beat the rule classifier (0.510 against 0.463; 0.448 at the old default). The action-safety thresholds keep their vendor values because the action answer scored below its constant baseline. The guardrail and action-safety consumers are documented as monitor-only on that endpoint, and routing as shadow-only for Korean traffic. tests/systemone_calibration_test.rs re-derives every default from the committed run and fails if the library or either language's defaults table disagrees. In mode: enforce this blocks less on the defaults: a violence probability in [0.70, 0.75) now flags instead of blocking (still a block at severity >= 2), and a category probability in [0.35, 0.40) no longer flags; set review_threshold and category_thresholds explicitly to keep the earlier bands.

• Accept GPT Image custom sizes and up to 10 images per request (#1740). Image generation refused every GPT Image request with n above 1 and every size outside 1024x1024, 1536x1024, 1024x1536, and auto, and image edits refused the same sizes, although OpenAI accepts n from 1 to 10 and, on GPT Image 2 and the GPT Image 2.5 models, any WIDTHxHEIGHT whose edges are multiples of 16, whose long edge is at most 3840, whose aspect ratio is within 1:3 to 3:1, and whose pixel count is within 655,360 to 8,294,400. One validate_gpt_image_size now serves generation and edits, names the rule a rejected size broke, and keeps the preset-only list for the GPT Image 1 models. n is 1 to 10 for GPT Image models on generation (edits already allowed it), DALL-E 3 stays at 1, and the catalog max_n for every GPT Image model becomes 10.

Removed¶

• Drop gpt-4-32k, gpt-3.5-turbo-16k, and o1-mini from the OpenAI backend's advertised model list and from the built-in OpenAI catalog, and move the o1-mini catalog entry to model-metadata-retired.yaml (#1739). Each is shut down on both the OpenAI API and Azure OpenAI (gpt-4-32k 2025-06-06, gpt-3.5-turbo-16k 2024-09-13 and 2025-04-30, o1-mini 2025-10-27 and 2025-11). Ids with a scheduled but future shutdown, such as gpt-4, gpt-4-turbo, gpt-3.5-turbo, o1, o3-mini, and gpt-image-1, stay listed until that date passes. Request handling for these ids is unchanged.

Fixed¶

• Fix deploy/docker-compose.systemone.yml so the service actually runs laya-serve (#1725). laya, which the inference extra installs, ships its own laya-serve console script that replaced the package's, so the container served /health instead of /healthz (the healthcheck never passed) and ignored the LAYA_SERVE_* settings. The compose file also turned an unset LAYA_SERVE_API_KEY into an empty string, which laya-serve treats as a configured key, so every request answered 401. The service now starts python -m laya_serve.cli and unsets an empty key.
• Report Claude's real context window and output cap from Anthropic and Bedrock model discovery (#1741). Both backends derived ModelInfo limits from a substring match that put every Opus at 200K / 32K and every Sonnet at 200K / 64K, so a native type: anthropic or type: bedrock backend, which never reads model-metadata.yaml, advertised 200K for Opus 4.6 through 5.5 and Sonnet 4.6 and 5, which accept 1M. A shared claude_model_limits now parses the family version (aliases, dated snapshots, and Bedrock inference-profile ids included), matches the catalog for every Claude entry under a test, and lets a later minor inherit its family's limits. Each backend's max_tokens_per_request capability moves from 200K to 1M.
• Bound the ZMQ frame length before allocation in the Dynamo kv_metrics subscriber (#1733, part of epic #1703). The SUB socket is now the router's own minimal ZMTP 3.x client (src/dynamo/load/zmtp.rs) over a plain tokio::net::TcpStream instead of zeromq, whose ZmqCodec::decode reads an 8-byte frame length off the wire and reserves that much capacity before a single payload byte arrives, exposes no maximum-message option, keeps its codec module private, and has no pluggable transport. Because the router dials endpoints that fleet-authored etcd records announce and [profile.release] sets panic = "abort", a hostile peer could end the whole router process from a 9-byte header without transmitting the bytes it declared, and could chain more-flagged frames to grow one multipart message without bound; the router's own 16 KiB payload check ran on an already-received frame and never reached it. The new client refuses a declared message-frame length above 32 KiB (compared as a u64 before any narrowing, so a declared u64::MAX cannot truncate into an acceptable number), a command frame above 4 KiB, a more-flagged chain past 8 frames against a wire message of exactly four, and a chain totalling more than 64 KiB — each checked before anything is sized from the wire value, with the read buffer allocated once from the compiled-in constant. It also refuses a peer that is not speaking ZMTP 3.x with the NULL mechanism to a PUB or XPUB socket. A refusal is counted once under the new continuum_dynamo_load_events_total{result="transport_refused"} label and drops the connection into the existing jittered reconnect, so a hostile or broken publisher can at most lose its own subscription, and the trust-boundary caveat that asked operators to treat dynamo.discovery.etcd writers as router-equivalent for process integrity is removed from the module doc and both architecture documents. The subscription lifecycle, connect budget, decoder, and store are unchanged. The reconnect schedule now restarts from its 500ms floor only after a session that stayed up for at least 30 seconds, rather than whenever a connect succeeded: a peer that completes the handshake and then fails at once would otherwise have held the schedule at the floor, redialling, logging, and counting several times a second per publisher for as long as its etcd record existed, and no signal the peer produces can decide this because a well-formed sample is one such a peer can also send. zeromq moved to [dev-dependencies], so no runtime graph resolves it and a release binary drops it, while tests/dynamo_load_zmq_test.rs still runs a real PubSocket as the publisher — the check that keeps the bounded client's greeting, READY exchange, and ZMTP 3.0 subscription form matching the crate Dynamo's event plane runs on.

Dependencies¶

• Float the dependency lock (47 packages, including rmcp 3.3.0 to 3.4.1) and name rmcp's InitializeResult directly instead of the ServerInfo alias that rmcp 3.4 deprecates (PR #1686). This supersedes the dependabot group update in PR #1683.

v1.30.0 - 2026-09-24¶

Added¶

• Add explicit endpoint/deployment UUID identities to Hub backend inventory and AppProxy ROUTER replicas without changing accounting attribution; expose them in backend configuration, Admin, probes, and schema (#1732).

• Add the shared System One decision-model client (src/services/systemone/, #1720, epic #1719): typed noul / choice / score questions over a state, decoded against the questions asked; one POST /v1/systemone per call with at most one retry on 429, 529, or a connection failure that honors retry-after inside the call's own timeout, and never a retry on 422 or a parsed response; a closed error enum so each consumer applies its own fail policy; error bodies capped at 200 bytes; the server's resolved model and usage surfaced for audit and internal usage accounting; systemone_requests_total{consumer, result} and systemone_request_duration_seconds{consumer} with no state or question text in any log or metric. The target is described once by the always-compiled SystemOneTargetConfig block (endpoint + api_key_env, or backend under the pinned-backend contract, plus model, timeout_ms, max_state_bytes, count_toward_budgets, max_concurrent), validated at load and documented in the configuration assistant; the consumer sub-issues embed it unchanged.

• Add the models released since the last catalog update to model-metadata.yaml, with published context windows, output caps, knowledge cutoffs, and per-MTok or per-image pricing. OpenAI: gpt-6-sol and gpt-6-luna (September 22) and the two GPT Image 2.5 models, gpt-image-2.5-sunburst and gpt-image-2.5-flare (September 8). Anthropic: claude-opus-5-5 (September 22). Meta: muse-spark-1.1, muse-spark-1.2, and muse-spark-1.3 with the lower-priced -contributor ids for 1.2 and 1.3, muse-image-1.0, and muse-voice-transcribe-1.0. Google: gemini-3.8-live, gemini-3.8-live-extended-thinking, gemini-3.5-transcribe, gemini-3.5-transcribe-live, gemini-robotics-er-2-preview, and gemini-3.1-flash-lite-image. Open-weight and API models from Chinese labs: deepseek-v4.1-flash (API id deepseek-flash), glm-5.3, glm-5.3-flash, glm-5.3-flashx, mimo-v2.6-pro, mimo-v2.6-flash, Tencent hy4-preview and hy3, Meituan longcat-2.0, Ant Group ling-3.0-flash, Shanghai AI Laboratory intern-s2-397b, StepFun step-3.7-flash, qwen-image-2.1, qwen-image-3.0-pro, and ByteDance dola-seedream-5-0-pro-260628 and dola-seedream-5-0-flash-260915. Existing entries record what changed upstream: deepseek-v4-pro carries the 0813 GA snapshot and the peak rate DeepSeek has charged since 2026-08-16, deepseek-v4-flash notes that its API id now routes to V4.1 Flash, gemini-3-pro-image-preview becomes the GA gemini-3-pro-image (the preview id stays as an alias) with pricing corrected to Google's token rate, muse-glimmer-30b drops the video input that neither its model card nor Meta's docs list, text-embedding-004 notes that the Gemini API shut it down while Vertex AI serves it until 2027-04-01, gpt-image-1, gpt-image-1.5, and gpt-image-1-mini give their scheduled shutdown dates, and the Gemini 2.5 models note that since 2026-09-18 Google serves them only to projects that already used them.

• Register gpt-6-sol and gpt-6-luna in the built-in OpenAI catalog as well, since a native type: openai backend's /v1/models discovery never reads the YAML. Unlike gpt-6-astra they accept reasoning_effort: "none" (none through max, default medium), and OpenAI documents that Chat Completions takes function tools only at none, so both carry the same tools-plus-reasoning bridge flags as the gpt-5.6 family: a tool request at none stays on /v1/chat/completions, and one at any other effort, or with the effort omitted, is bridged to /v1/responses. The two GPT Image 2.5 models are registered in the built-in image catalog, and image generation accepts their new xhigh and max qualities, which the router still refuses for earlier GPT Image models.

• Support Claude Opus 5.5's request surface. Opus 5.5 keeps thinking always on and returns HTTP 400 for an explicit disabled-thinking config at every effort, so it now takes the Mythos-class omission path (model_rejects_disabled_thinking) instead of the Opus 5 rule that accepts the config up to effort high; the two gates still never fire for the same id. It also removed forced tool use, so model_rejects_forced_tool_choice now covers Opus >= 5.5 alongside Mythos-class >= 5.1, and the router answers a forced tool_choice for it with its own 400 naming the documented remedies on all three ingresses. Both gates are parsed from the family version, so Opus 5 keeps its current behavior and later Opus minors inherit the new rules.

• Stamp what each usage record's ttft_ms measured: every control-plane usage producer now names the span with an additive ttft_kind protocol field (TtftKind: streamed_first_token, response_headers, cache_replay) carried through UsageEvent and into_usage_record (#1641). The field is set only when ttft_ms is set, and a record whose ttft_ms is absent never carries one. This is an additive local protocol change ahead of the upstream re-sync; the hub-side analytics fix is continuum-hub#1164.

Changed¶

• Move every default that pointed at a shut-down or restricted model to a model a new project can call. An image generation request without model now defaults to gpt-image-2 instead of dall-e-3, an image edit to gpt-image-2 instead of gpt-image-1 (which OpenAI shuts down on 2026-10-23), and an image variation to nano-banana instead of dall-e-2, OpenAI's only variations model; dall-e-2 and dall-e-3 were shut down on 2026-05-12, so the old defaults sent every such request to a retired model. GPT Image models reject response_format, so a request that omits model but sends response_format now gets the router's usual 400 for that combination. nano-banana now maps to gemini-3.1-flash-image instead of gemini-2.5-flash-image, which Google limited to projects that already used it on 2026-09-18, and the catalog alias moves with it; naming gemini-2.5-flash-image directly still passes through. The Gemini backend's fallback model list, used when discovery fails, becomes gemini-3.1-pro-preview, gemini-3.8-flash, gemini-3.5-flash, and gemini-3.5-flash-lite, and its advertised model list drops gemini-2.0-flash, gemini-2.0-flash-exp, gemini-2.0-flash-lite, text-embedding-004, and imagen-3.0-generate-002 in favor of gemini-3.1-flash-image and gemini-3-pro-image. The OpenAI backend's advertised list drops dall-e-2 and dall-e-3 and adds gpt-image-2 and the two GPT Image 2.5 models. The default timeouts.limits.extended_models list adds gpt-6-astra, gpt-6-sol, gpt-6-luna, claude-fable-5-1, and claude-opus-5-5 and drops gemini-3-pro, so a per-model streaming override above max_streaming_timeout is accepted for the new frontier tiers and refused for the retired one; an operator list still replaces the default wholesale.

Removed¶

• Move the metadata of models shut down on every serving surface out of model-metadata.yaml into a new model-metadata-retired.yaml, which the router does not load: dall-e-2, dall-e-3, gemini-3-pro (with its gemini-3-pro-preview alias), gemini-2.0-flash, gemini-2.0-flash-lite, imagen-4, and imagen-3.0-generate-002. Each was checked against the provider's own API, Azure OpenAI, and Vertex AI on 2026-09-23, and each entry records where and when it was shut down; text-embedding-004 stays because Vertex AI still serves it. The file keeps the base catalog's schema, so a deployment that still serves one of these ids restores its metadata by copying the entries into a model-metadata.d/ drop-in. dall-e-2 and dall-e-3 also leave the built-in OpenAI catalog, following the rule that a retired model is absent from both catalogs; request handling for DALL-E ids is unchanged. A new test fails the build if the retired file stops parsing or validating, or if an id or alias appears in both files, and the image metadata tests read both files so the DALL-E-only per_image and limits shapes stay covered.

Fixed¶

• Accept gpt-image-2, gpt-image-2.5-sunburst, and gpt-image-2.5-flare on /v1/images/edits. The edit endpoint's model allowlist stopped at the GPT Image 1 generation, so the router answered every edit for a current GPT Image model with a 400 before it reached a backend.

• Map nano-banana-pro to Google's GA gemini-3-pro-image id instead of gemini-3-pro-image-preview, which Google shut down on 2026-06-25. Every nano-banana-pro image generation or variation request was reaching a retired model.

• Repair the disaggregated prefill/decode orchestrator scaffold so real workers can land on it (#1698, part of epic #1703). The fast-decode lookup now keys the KV cache index the way the KV listener does, with the 64-hex SHA256 prefix key from the shared prefix_key_hex encoder, gated on prefix_routing.enabled like the KV overlap scorer; it used to hash the request with DefaultHasher into 16 hex characters, a keyspace no real event ever populated. A stream: true request now consults the orchestrator: the decision is taken ahead of the streaming split, the selected backend goes into the streaming pipeline as a one-element allow-list, and the x-continuum-routing-path and backend headers are stamped on the streaming response, with fallback_to_unified re-dispatching a failed routed attempt through unified streaming and the 503 envelope preserved when it is off. BackendInfo.healthy is the health checker's verdict and in_flight the pool's live gauge instead of the hard-coded true and 0, and the freshest engine-stats waiting_requests and kv_cache_usage ride along for the picker. Role filtering goes through the pool's registered RoleFilterScorer (BackendPool::phase_eligible_candidates), which reads roles from a new per-pool configuration identity map; BackendPool::update_from_config diffs that identity instead of (name, url) only, so a role flip is re-read in place with no drain (role also joins the hot-reload backend change detection, which is what lets that reconcile run at all for a role-only edit), a models, weight, or request_extensions change drains and rebuilds the backend while keeping its engine-stats snapshot, and a URL change keeps its existing resets. The disaggregated_* metric family is registered in the global registry and written by the orchestration arms: disaggregated_requests_total{routing_path}, disaggregated_fallback_total, and disaggregated_errors_total{phase}, with the duration histograms registered but reserved for the picker. The architecture page now describes the KV transfer as engine-owned (NIXL between workers) with the router brokering selection only, and documents external_storage as a legacy field. The public continuum_router::services::disaggregated::BackendInfo struct changed: in_flight is now u64, waiting_requests and kv_cache_usage are new fields, and a BackendInfo::new constructor was added; a downstream caller building BackendInfo values directly should review its call sites.

• Clamp TokenUsage::from_json_value's cache-subset fields to the inclusive prompt count instead of storing a provider-reported value verbatim (#1678). A cached count above prompt_tokens, such as OpenAI prompt_tokens_details.cached_tokens, Responses input_tokens_details.cached_tokens, a top-level cache_read_input_tokens, or Gemini's cachedContentTokenCount, was stored verbatim, so the Hub record's cached_input_tokens could exceed its inclusive prompt. A new clamp_cache_subsets step clamps cache_read_input_tokens to prompt_tokens first and cache_creation_input_tokens to the remainder, leaving None as None and logging nothing. Router-local billable input and the Hub record's input_tokens are unchanged for every shape; only an over-reported cached bucket shrinks. The streaming /v1/responses tap now builds its usage by parsing through from_json_value instead of constructing TokenUsage by hand, so it reaches the same clamp.

v1.29.0 - 2026-09-21¶

Added¶

• Add searxng as a web_search.provider, so the router-managed web_search tool can run against a self-hosted SearXNG instance instead of a hosted search API (#1664). The three existing providers (Serper, Exa, Brave) each need a commercial API key and a route to the public internet, which an air-gapped or policy-restricted deployment may not have; base_url (#803) could point them at a mirror but not at a different engine. The new SearxngProvider implements the same SearchProvider trait behind build_provider, so the injection loop, the Anthropic web_search_20250305 emulation, metrics (provider="searxng"), and result sanitization are unchanged. It calls GET <base_url>?q=<query>&format=json with no credential and maps each entry of the JSON results array (url, title, content, any of which SearXNG may serialize as null) to a result, truncated client-side to max_results because SearXNG has no per-request count parameter. Two things differ from the hosted providers and are enforced at load, hot reload, and config validate: SearXNG has no public endpoint, so base_url is required and names the instance's /search endpoint with no query string or fragment (the same rule Brave already had, since both append their own query string), and it has no API key concept, so api_key is ignored and never forwarded. Two instance-side settings are the usual failure modes and are documented as such: an instance whose settings.yml does not list json under search.formats answers every search with HTTP 403, and server.limiter: true classifies the router's non-browser requests as bot traffic and answers 429 unless the router's address is in botdetection.ip_lists.pass_ip; both surface as the upstream status (429 maps to the Anthropic too_many_requests code). A 200 with no results whose body also names unresponsive_engines is logged at warn with the engine names, since an instance whose engines cannot reach the internet answers every query that way. A configuration that names one of the existing providers loads, validates, and dispatches exactly as before. The configuration-assistant skill document moves to schema 1.56.0.

• Let an operator declare the reasoning_effort vocabulary a served model accepts (#1545). Nothing on an OpenAI-compatible wire says which levels a self-hosted engine takes, so the router fitted the level it derived onto one hard-coded spelling per backend type (low, medium, and high, with both high and xhigh becoming high for every self-hosted engine type), and forwarded it unchanged. A backend whose vocabulary excludes high therefore answered 400 for a client that asked for high, and there was no way to tell the router what that engine takes: BackendConfig had no effort field, request_params is a closed set of seven typed numeric sampling parameters, and no serving engine documents a per-model accepted-effort set. A new optional metadata.reasoning_efforts array on a model entry now names the spellings that model accepts (minimal, low, medium, high, xhigh, max, in that order), and the closest declared level is used instead, by the same nearest-level search and tie-break the built-in OpenAI model table already applies. max and minimal are spellings the router never requests, so a declaration is the only way they become reachable; none is not a level and is refused at load with a message naming rejects_reasoning_effort_none; and a declaration that answers nothing for the requested level (empty, or holding an entry the fitter cannot use) is treated as no declaration rather than as a vocabulary that accepts nothing, so a hot-reloaded configuration carrying one cannot silently strip an effort from a hop the operator only meant to narrow. The declaration is resolved from the selected backend's own model_configs first and the shared metadata cache second, so a model-metadata.d/ drop-in can declare one model's vocabulary for every backend that serves it and can override a built-in OpenAI table entry without a release; the declaration is read from the snapshot the request was routed against at every configured dispatch site, including each mid-stream fallback hop; the typed Gemini transform reads no declaration at all, because it pins its backend type and holds no configuration. Fitting runs once per attempt against the selected backend, on the Chat Completions wire, on the Responses wire, and in the dispatch-side conversion that serves a client sending Anthropic-native thinking to /v1/chat/completions; because it is per attempt, a fallback hop fits against its own target and never against the previous one's declaration. On the Anthropic Messages ingress the value being fitted is the router's own: AnthropicMessagesRequest has no #[serde(flatten)] catch-all and CanonicalParam has no reasoning variant, so a client cannot put the field there at all, which is why rewriting it leaves the documented invariant that a client-supplied effort is never overwritten intact. A client that sends reasoning_effort directly to /v1/chat/completions therefore keeps exactly what it sent: fitting the OpenAI-wire direct path would overwrite a genuinely client-supplied value and is a separate decision this change does not make. A declared vocabulary is intersected with Gemini's own VALID_REASONING_EFFORTS, because that backend's transform runs after the fit and would otherwise remove the field silently and leave the request at Gemini's default effort. A declaration on backends[].model_configs[] is a load error when it names an unknown or duplicate entry, matching every other operator-authored backend field, while the shared metadata layer stays warn-only, because a downloaded catalog must not be able to brick a reload. An unset declaration changes nothing: every payload is byte-identical to before.

• Add per-backend request_extensions so an operator can set an OpenAI-compatible aggregator's routing preferences and attribution once in configuration instead of in every client request (#1631). A client could already send OpenRouter's provider, models, and transforms fields and its HTTP-Referer/X-Title headers, but there was no per-backend place to set a house policy, and a client that cannot be modified could not have one: request_params is a closed set of seven typed sampling parameters by design, and BackendConfig had no extra-headers field. request_extensions.body.defaults fills a key only when the client left it unset (client wins) and request_extensions.body.overrides replaces the client's value (config wins), both as a deep merge of JSON objects in which arrays, scalars, and type conflicts are decided wholesale at the key, so overrides.provider.data_collection: deny enforces one leaf while keeping the client's provider.sort. request_extensions.headers adds headers that replace a forwarded client header of the same name. The fragment is merged last into the fully built wire payload of the selected backend at every send site, on every retry and fallback hop (a hop to another backend carries that backend's extensions and never the previous one's), for chat completions, completions, responses, and embeddings, including the Anthropic Messages and Responses ingresses converted for an OpenAI-wire backend; headers also ride on model discovery, health probes, pre-warm, control-plane probes, and the realtime handshake. Extended in #1652 to one rule: every request that carries the backend's credential carries its extension headers. The guardrail backend: and smart-routing classifier calls through the typed backend executor, provider batch lifecycle calls, engine-statistics scrapes, backend type detection, Gemini context-cache management, the model auto-discovery a typed backend runs at construction, llama.cpp /props capability detection, and the typed Anthropic and Gemini connection pre-warm now carry them too, over both HTTP and Unix socket where the site has both. A consumer that outlives one configuration generation reads the block per send, so an Admin backend edit or a config-sync change reaches its next call, and a headers-only edit restarts an engine-statistics poll task so the next scrape carries the new values. The Admin backend-create type probe runs before the candidate is validated, so it attaches the candidate's headers only when they already pass header validation and otherwise probes without them, which keeps an unvalidated Authorization extension from replacing the key on the wire. The Unix-socket health probe still sends neither the credential nor the headers. Validation runs at load on every ingestion path with a named path in the error: body is a load error on anthropic, gemini, and bedrock, and headers on a Bedrock runtime/converse backend, whose SigV4-signed dispatch cannot carry them; request identity, stream, the request_params keys and their max_completion_tokens/max_output_tokens aliases, tool and output-shape fields, the response-shape fields text, logprobs, top_logprobs, encoding_format, and dimensions, router-managed fields, and any key the dispatch strips for that backend (including stop on a Codex OAuth backend) are reserved, while stop, seed, and logit_bias otherwise stay settable; auth-owned and router-owned header names are refused case-insensitively and with _ read as -; and the fragment (16 KiB, depth 8), header count (16), and header name and value sizes are bounded, with control characters refused. Tightened in #1650: background, include, include_reasoning, prompt_cache_key, and safety_identifier (Responses), modalities, audio, and prediction (Chat Completions), the legacy Completions echo, vLLM's prompt_logprobs, and vLLM's structured-output fields guided_json, guided_regex, guided_choice, guided_grammar, and their vLLM 0.12 replacement structured_outputs are now reserved for the same reason as the fields above, since each would change request content or the response shape a client parses, while service_tier, prompt_cache_retention, and prompt_cache_options stay settable alongside stop, seed, and logit_bias. Header values must also be visible ASCII (SP through ~): a value with a byte outside that range, such as café or Hangul, is now refused at load, because the wire client otherwise sends it as RFC 9110 obs-text that upstreams and intermediaries handle inconsistently. request_extensions has not shipped in a release yet, so this is a same-cycle tightening rather than a migration: a configuration written against the block as #1631 first added it, using one of these now-reserved body keys or a header value that is not visible ASCII, fails to load under this and later builds; drop the key or percent-encode the header value. Header values support ${ENV_VAR} and are treated as secrets whatever the header is called: masked on the Admin config and backend reads, export, history, and the WebUI with names left visible, preserved across a masked GET-then-PUT round trip, redacted by config diff and the MCP tools, never logged, and reported to Continuum Hub only as the request_extensions unsupported-field code on a local backend export. A change applies to the next request without a restart, a backend with a body fragment keys its response-cache entries by a fingerprint of the fragment so an edited fragment is never answered from entries stored under the previous one, and a configuration without the field serializes and dispatches exactly as before.

• Add model_experiments, so one client-facing model name resolves per request to one of several weighted variants, each naming an upstream model and optionally pinning backends (#1634). Resolution runs inside dispatch_model_for_request, so every ingress that already dispatches aliases gets the same ordering, assignment is sticky and deterministic across replicas, attribution is recorded per variant, and one arm never answers from another arm's cache. Up to 32 experiments each carry up to 16 variants; sticky_key selects the assignment key (api_key by default, or user, none, or header:<name>), sticky_ttl bounds an assignment's lifetime between 60s and 365d, and hide_variant_models reserves the variant models for experiment traffic by hiding them from listings and answering a direct request for one with the ordinary 404.

• Add GET /admin/control-plane/policy, a read-only view of the Continuum Hub policy the router is executing rather than the one the Hub last sent (#1451). It reports tier limits with per-tier key counts, substitution rules, equivalence classes, routing windows, the optimization policy, model-budget pool state, and the status reported back to the Hub, built field by field from one policy snapshot so key hashes and provider credentials never reach it and keys appear only as counts. It is mounted beside /admin/control-plane/status under the same control-plane gate, with a Hub policy card on the Integrations page covering the not-yet-synced, hub-unreachable, and rejected-revision states.

Fixed¶

• Enforce rate_limiting.limits.per_backend as a selection-time candidate filter, and stop shipping it enabled by default (#1682). The dimension was evaluated by the identity admission mount, which read a RateLimitBackendName request extension nothing ever inserted, so every request took the "no backend pinned" branch and the limit never charged; meanwhile the shipped default carried per_backend: 100 req/s, burst 200 on every deployment that never configured rate_limiting.limits, so simply making it work would have started refusing traffic on upgrade. A per-backend limit is a capacity constraint on that backend, which makes it a routing input rather than a refusal reason: the shared selection seam now drops a backend whose bucket is empty from the candidate set, beside the health and circuit-breaker filters, so the request goes to another backend serving the same model, and the bucket is charged once at the dispatch boundary, so a backend the filter dropped, a backend the strategy did not pick, a backend a fallback hop moved away from, a cache hit, and a lost half-open admission race are all uncharged. Only when every candidate is exhausted is the request refused, with a 429 carrying error.details.limit_type: "backend" and a retry-after the bucket can honor, distinct from the 503 service_unavailable an outage produces; with a fallback chain, the refusal is the 429 trigger, so the chain advances to a model whose backends have room, and a chain that exhausts on it answers the 429 rather than the generic exhaustion 500. This applies on every dispatch path (chat, streaming and its mid-stream relay, Responses, Anthropic Messages and count_tokens, images, realtime, ACP), a source audit pins that no path opens circuit admission without charging, and the admission-time path (RateLimitBackendName, its read, the per-mount flag, and the skip accounting) is removed rather than left beside the filter. The shipped default becomes per_backend: None, matching per_api_key, so no deployment's live enforcement changes: the dimension never charged before, and it now enforces only where an operator configures it. Under storage: redis the bucket is shared across replicas and degrades to the replica's in-memory bucket when Redis is unreachable, as the other dimensions do; whitelist and bypass_keys exempt callers from the admission mounts and do not exempt a backend from its own capacity.

• Report streamed Anthropic usage to /v1/chat/completions clients and to the router's own accounting in OpenAI shape (#1640). The stream transformer copied message_delta.usage verbatim onto the finish chunk, so a client received {"output_tokens": N} with no prompt_tokens, completion_tokens, or total_tokens, and the input and cache counts, which Anthropic reports only on message_start, were dropped entirely. The usage parser then found no prompt count and recorded prompt_tokens: 0 for Prometheus, admin statistics, hub TPM, and the control-plane usage tap, so a Backend.AI GO agent run on an Anthropic provider metered no streamed input at all. The transformer now accumulates the counts from message_start and every message_delta (Anthropic reports cumulative values, so a field present on a later event replaces the earlier one, keeping only the four known counters and only when they are unsigned integers, so a garbage value on a later event cannot erase a count an earlier one reported) and emits one usage-only chunk at message_stop with choices: [] and a usage mapped by the same helper the non-streaming transform uses, which is the shape OpenAI sends under stream_options.include_usage and that Bedrock Converse already emitted. The chunk is emitted whatever the client asked for in stream_options, because the router's own taps read it. The finish chunk no longer carries usage. This covers every caller of the shared transformer: native Anthropic over HTTP and over a Unix socket, type: anthropic backends, and Bedrock endpoint_type: runtime and mantle. On a cache hit the streamed prompt_tokens is now the inclusive count (non-cached input plus cache reads plus cache writes) with prompt_tokens_details.cached_tokens alongside it, matching what #1642 made the non-streaming path report.

• Serialize the /v1/responses usage breakdown under the OpenAI Responses API field names input_tokens_details and output_tokens_details (#1660). The router emitted the singular input_token_details and output_token_details, which no client following the OpenAI schema looks for, so the cached-token count was invisible to every such consumer and any of them that prices cached input separately billed the whole prompt at the full input rate. This is a client-visible rename on /v1/responses: the non-streaming body, every response-carrying streaming event (response.created, response.completed, and the rest), and a stored session now all carry the plural keys, and the singular ones are gone from every body this router serializes itself. The two bodies it does not serialize are the exception: a /v1/responses cache hit replays the stored bytes verbatim, so an entry written before the upgrade keeps the singular keys until it expires, and a pass-through body is forwarded exactly as the upstream sent it, so an upstream still on an older build supplies the singular keys. Metering is unaffected in both cases because the parser accepts either spelling. The member names inside the two objects are unchanged, because cached_tokens and reasoning_tokens already match the OpenAI schema and the remaining members (cache_write_tokens, text_tokens, image_tokens, audio_tokens) are router extensions that schema does not define. Both spellings still decode on input, so a mixed-version pair of routers keeps metering correctly in both directions: a newer router reading an older peer's singular body, an older router reading a newer peer's plural body (the reads added in #1645 already preferred the plural spelling), and a cached or stored Responses body written before the upgrade all keep the cache read as the Hub record's cached_input_tokens instead of billing it at the input rate. A downstream consumer that parses this router's /v1/responses usage and looks for the singular key has to be updated; one that follows the OpenAI schema needs no change and sees the breakdown for the first time.

• Stop charging cached prompt tokens twice on Continuum Hub usage records (#1642). The Hub prices input_tokens and cached_input_tokens as disjoint buckets, but the record sent the backend's prompt_tokens as input_tokens, and for OpenAI-shape responses that count already includes prompt_tokens_details.cached_tokens, so every cached token was billed at the input rate and again at the cached rate; local cache replays were billed the same way. prompt_tokens is now the inclusive prompt count for every producer (non-cached input plus cache reads plus cache writes) and the record carries input_tokens = prompt_tokens - cache reads, the same billable input router-local input_tpm and monthly enforcement already charged. Shapes that report an exclusive count are converted where they enter: the usage parser adds cache_read_input_tokens and cache_creation_input_tokens back onto an Anthropic-native input_tokens, the native /v1/messages stream tracker now also captures cache writes, and the Anthropic and Bedrock Converse transforms to /v1/chat/completions emit the inclusive prompt_tokens with prompt_tokens_details.cached_tokens and keep the top-level Anthropic cache keys even without a thinking block. Two visible consequences: prompt_tokens returned to /v1/chat/completions clients on an Anthropic or Bedrock cache hit is larger than before, and on those cache hits router-local input TPM and monthly token counters now charge the non-cached input plus cache writes (before, the cache read was subtracted from Anthropic's already-exclusive count, which could zero the charge), while Prometheus input-token counters, admin statistics, model experiment token totals, and capacity observations now count Anthropic and Bedrock cache reads and writes as prompt tokens. Anthropic streaming on /v1/chat/completions reports its prompt usage from #1640, in the same release. The reverse bridges follow the Anthropic convention: when /anthropic/v1/messages is served by an OpenAI-compatible or Responses API backend, the non-streaming body and the streaming message_start/message_delta usage now report input_tokens as the upstream prompt minus cache reads and writes (reading prompt_tokens_details.cached_tokens and input_tokens_details.cached_tokens as well as the top-level keys), so a client, or this same router re-parsing the bridged body (a non-streaming response, its own local cache replay, or a bridged stream it reconstructs itself), no longer counts the cached prompt twice; on a cache hit those clients see a smaller input_tokens. The native /v1/messages streaming tracker now also reads input_tokens, cache_read_input_tokens, and cache_creation_input_tokens from message_delta (the Anthropic Messages API repeats those cumulative counts there, and a bridged stream defers them there because its message_start carries input_tokens: 0), so a router metering another router's bridged stream records the real prompt instead of 0. /v1/responses served by an Anthropic backend now reports the Responses-convention inclusive input_tokens with input_tokens_details.cached_tokens and cache_write_tokens on both the non-streaming and streaming paths (streaming previously recorded Anthropic's exclusive count, or 0, and dropped both cache buckets), /v1/responses served by an OpenAI-compatible chat backend keeps prompt_tokens_details.cached_tokens as input_tokens_details.cached_tokens on the streaming path, and the usage parser reads that spelling as well as the singular input_token_details this router emitted before #1660, so a router metering another router's /v1/responses body, or replaying its own cached one, keeps the cached bucket instead of billing it at the input rate.

• Decode an OpenAI-compatible /v1/models catalog that omits the decorative object and owned_by fields, instead of failing the whole catalog with ModelFetchError::ParseError (#1629). ModelsResponse.object, Model.object, and Model.owned_by had no serde(default), so an aggregator such as OpenRouter, whose catalog carries none of the three, was classified as a permanent parse failure and left GET /v1/models empty for that backend forever, even though the same backend served chat traffic normally. object now defaults to "list"/"model" and an absent owned_by decodes as an empty string, which now reaches the owned_by repair in process_models_response (an openai backend fills it with "openai", except that a vendor/model id takes its vendor namespace, see #1630) instead of being rejected before that repair ever runs. An explicit JSON null on object, owned_by, or created, which servers that serialize unset optional fields as null emit, now decodes the same as an absent field instead of failing the catalog with a type error. id stays required, so an entry with a missing or null id still fails to decode. Both strict decode sites (handle_successful_response over HTTP and handle_unix_socket_response over a Unix socket) now share one decode_models_response helper, so a future change to this logic cannot fix one transport while missing the other. Making those fields optional also shrank the smallest accepted catalog entry from 36 bytes to 10 ({"id":""}), so a catalog inside the default 10 MiB max_response_size could decode to roughly a million entries before max_models_per_backend truncated it, a measured 657 MB peak per fetch against 175 MB for the densest catalog accepted before, and several such backends stack because fetch_all_models holds every result until the whole fetch pass finishes. Both decode sites now cap entries during decode itself instead of truncating afterward, keeping the first max_models_per_backend entries and still failing the catalog on a malformed entry past the cap, which brings the same body down to a 3 MB peak.

• Report an aggregator backend's model vendor and its upstream pricing, context window, and output cap in the model catalog (#1630). An OpenAI-compatible aggregator such as OpenRouter sends no owned_by on any catalog entry and namespaces every id by vendor, so the backend-type substitution in process_models_response stamped every model it listed, from anthropic/claude-haiku-4.5 to deepseek/deepseek-v4-flash, as owned_by: "openai", and the router ignored the context_length, top_provider.max_completion_tokens, and pricing the upstream sent, so each of those models read as unpriced with no known context window. When an upstream sends no owned_by for a vendor/model id, the first path segment is now the vendor. A real upstream value is still kept, and an explicit placeholder such as OpenAI's "system" still becomes the backend-type owner, as does a missing value on an id without a namespace; a per-model metadata owned_by still wins, and response_defaults.owned_by still applies only to a value that remains a placeholder. GET /v1/models keeps its OpenAI field set and changes only in that vendor. On GET /v1/models/extended and GET /v1/models/{model}, context_length joins vLLM's max_model_len as a context-window spelling (max_model_len wins when both are present), top_provider.max_completion_tokens sets limits.max_output, and therefore max_tokens, when it is within the entry's context length, and pricing.prompt/pricing.completion are converted from USD per token to pricing.input_tokens/pricing.output_tokens in USD per 1M tokens, with input_cache_read becoming cached_input_discount; the raw upstream fields keep passing through on the extended listing. Upstream values are treated as untrusted: a price must parse to a finite, non-negative number no larger than 100,000 USD per 1M tokens, so OpenRouter's "-1" for a variable price and an unparseable value publish no price rather than zero. Configured metadata from model-metadata.yaml, a model-metadata.d/ drop-in, backend model_configs, or the built-in OpenAI catalog still wins for limits and every other field, but not for the price, which is specific to the provider serving the model: an upstream price wins over pricing from the shared model-metadata.yaml, its drop-ins, and the built-in catalog however the entry was matched, so a shipped 0/0 self-hosting entry reached through vendor/ prefix stripping no longer makes a model such as minimax/minimax-m2 read as free, and only a price set in the serving backend's own backends[].model_configs wins over the upstream one. When several backends serve one id, the context window and output cap are the minimum across them, and the price folds one candidate per backend (its model_configs price if it pins one, otherwise its upstream price) into the component-wise maximum with the smallest discount. Both are collected per backend during deduplication the way #1456 collects max_model_len, so they do not depend on pool order; the published price is the highest any backend states, and a backend whose price is missing or "-1" is left out of the fold rather than bounded by it. A model kept listed because its backend's discovery failed now has its metadata recomputed from current configuration and the retained per-backend reports on every refresh, instead of republishing the folded context window, output cap, and price cached from the pass before, which could still reflect a backend no longer serving the id (this also applies to the #1456 context window). Two pre-existing passthrough hazards on the extended listing are closed as well. An upstream entry carrying a key the router writes itself (backends, tier, domains, or metadata; a type: continuumrouter upstream's own listing emits tier and domains) put that key into one JSON object twice, and such keys are now dropped from the passthrough. An upstream metadata object was decoded as router-trusted metadata, which let an upstream publish unvalidated pricing and failed the whole catalog on a shape mismatch; it is no longer decoded.

• Accept model ids containing / on GET /v1/models/{model} and GET /admin/smart-routing/model-profiles/{model} (#1647). Both routes were mounted as a single-segment axum capture, so a namespaced id such as OpenRouter's anthropic/claude-haiku-4.5 or vLLM's HF-style Qwen/Qwen3-32B 404'd in raw form and only resolved percent-encoded (anthropic%2Fclaude-haiku-4.5), which PR #1644 had to document as a workaround. Both are now mounted as an axum catch-all ({*model}), so the raw and percent-encoded forms return identical bodies; the reserved static segments extended, proxy, and refresh on /v1/models keep routing to their own handlers ahead of the catch-all, unchanged. A new model_id_from_models_path helper in src/http/middleware/model_extractor.rs parses the same decoded id that axum hands the single-model handler, and EnhancedRateLimitMiddleware::extract_model now uses it (as does the unmounted extract_model_middleware); the rate limiter previously read path.split('/')[3] off the raw, undecoded path and so keyed the per-model rate-limit dimension on anthropic rather than anthropic/claude-haiku-4.5 for a namespaced id, and on the literal extended for /v1/models/extended.

• Scope request de-duplication to the configuration snapshot each request runs under (#1648). The de-duplication key hashed only the endpoint, the request body, and the inbound headers, and a hit returns the earlier result before any backend is selected or dispatched to, so a byte-identical non-streaming request repeating within retry.timeout of an earlier one replayed that earlier answer across a configuration publish: a hot reload, an Admin API write, a control-plane config-sync install, or an AppProxy reconcile that changed backends[].request_extensions, backend membership, a backend's models list, URL, or credential, or selection_strategy took effect for everything except such a repeat. Each key now folds in a per-process random salt and the address of the Arc<Config> snapshot the request executes under, and every entry keeps a Weak handle to that allocation, which reserves the address while an entry is keyed on it and still lets the superseded configuration's contents drop with its last request. Within one published snapshot nothing changes: identical requests still coalesce onto one upstream call, and a request waiting on an in-flight one still receives its result. Any publish, including one that changes nothing routing-relevant, now starts a fresh window, costing at most one extra upstream call per distinct request per publish. Streaming is unaffected, because it is not de-duplicated. Expired entries are now reclaimed by a periodic sweep started with the server, because an entry keyed on a superseded snapshot is never looked up again and so never reaches the lazy expiry check that a lookup performs; without it, eviction at max_entries would be the only reclaim path. In AppProxy worker and ROUTER modes the effective window is shorter than retry.timeout, because every reconcile tick republishes the configuration whether or not anything changed, making it appproxy.reconcile_interval (default 15s). Library surface, for anyone using the public continuum_router::services::deduplication module directly rather than the documented serve_embedded entry point: DeduplicationEntry gains a public config_snapshot: Weak<Config> field, and DeduplicationManager::{generate_request_hash, mark_in_flight, cache_success_with_backend, cache_error} and EnhancedRetryHandler::execute_with_deduplication_attributed each take the snapshot as a new leading parameter.

• Re-validate the composed configuration when runtime backends join a hot reload (#1649). A reload validated the file on its own and never the result of merging in the backends created through POST /admin/backends and kept in the backends_persistence_file sidecar, so the router could publish an effective configuration that the next startup refuses: a reloaded tracing.headers name that a runtime backend already uses in request_extensions.headers, or a renamed file backend that duplicates a runtime backend's backend_id. The reload worker now composes a candidate and runs the shared validation gate on it before anything is published, and a failure takes the existing retry-and-retain path, so the previously published revision stays live. The ledger write that records which runtime entries a file shadows is deferred until publication, so a rejected reload cannot make a live runtime backend report configured and lose the next Admin update to it on restart.

• Escape upstream-authored text before it reaches a model-discovery log line (#1651). Every configured backend's /v1/models is untrusted input, and tracing-subscriber escapes ANSI/ESC in a log message but not a raw \r/\n, so an id or owned_by value carrying one could forge a second log record; process_models_response and eighteen sites across models::aggregation interpolated model.id/model.owned_by directly into debug!/trace! messages, reachable today from any backend whose catalog entries a client or operator does not fully control. Two new helpers in core::text_utils, escape_upstream_text (bound and escape control characters plus U+2028/U+2029) and describe_upstream_json_error (apply that to a serde_json::Error's message while keeping its line/column), are now used at every such site and at the three /v1/models//props parse-error paths, closing both the forging vector and the unbounded log growth an oversized upstream value could cause on a parse failure.

• Surface Gemini's context-cache read on both native-shape usage conversions (#1659). Gemini's promptTokenCount is inclusive of usageMetadata.cachedContentTokenCount, exactly like OpenAI Chat Completions' prompt_tokens, but neither native-shape conversion surfaced the cached count, so every cached prompt token was billed at the full input rate and Hub cached_input_tokens stayed 0, the same under-reporting #1642 fixed for the OpenAI and Responses bridges. transform_native_gemini_response, what the /v1/chat/completions HTTP and Unix-socket relays apply when a Gemini-compatible backend answers in the native candidates shape, now emits prompt_tokens_details.cached_tokens. GeminiConverter::extract_usage, the /v1/responses ingress and the only one that calls :generateContent and :streamGenerateContent natively, streaming and non-streaming alike, now emits input_token_details.cached_tokens, and the non-streaming stats and control-plane tap carries it through as prompt_tokens_details.cached_tokens. So a Gemini response with promptTokenCount: 100000 and cachedContentTokenCount: 90000 records 10000 billable input tokens and 90000 cached instead of 100000 billable, and bridges to Anthropic input_tokens: 10000 with cache_read_input_tokens: 90000 while the router keeps charging the inclusive 100000 minus the cache read. The still-unwired direct Anthropic-to-Gemini bridge (gemini_transform::transform_gemini_usage, prepared for a future native routing path and not on any live request path today) gained the same fix and now reuses bridged_input_usage rather than a second implementation, and its two truncating promptTokenCount/candidatesTokenCount as u32 casts, along with the ones in GeminiConverter::extract_usage, are now the saturating conversion every other usage path from #1642 uses. Gemini has no cache-write concept, so cache_creation_input_tokens stays absent on every path. A type: gemini backend answering /anthropic/v1/messages or /v1/chat/completions is normally reached through Gemini's own OpenAI-compatible surface, which already reported usage.prompt_tokens_details.cached_tokens and already split correctly; that path is unchanged.

• Merge a request's system messages into one leading message for models whose chat template accepts nothing else, so Claude Code no longer fails from its second turn against a Qwen3.8 backend (#1663). /anthropic/v1/messages served by an OpenAI-compatible backend forwards a mid-conversation system message at the system role, in place, so the instruction keeps system priority, and the OpenAI ingress passes a client's own mid-array system entry through unchanged; every Qwen/Qwen3.8-* chat template raises System message must be at the beginning. for any system entry that is not the first message, which vLLM answers with HTTP 400 and the router relayed as 400 Invalid request sent to backend, so Claude Code, which sends such a message from its second turn on, failed every turn after the first against that backend. The check is the model's template, not the engine, and it also rejects a second system entry placed directly after the first, so the entries have to merge rather than move. Because the constraint ships with the model, it is model metadata: a new requires_single_leading_system key, set in the built-in catalog for the Qwen3.8 family (qwen3.8-27b, whose provisional entry is confirmed against the published weights, plus new qwen3.8-2.4t-a95b and qwen3.8-flash-next entries), which an HF-style served id such as Qwen/Qwen3.8-27B-FP8 reaches through the usual namespace-prefix stripping and FP-format peeling, so a deployment that serves the model under its Hugging Face name needs no configuration. A deployment that names the model itself declares the key in the backend's model_configs entry, the first source the metadata lookup consults; the key resolves field-wise across the sources, so an entry that leaves it unset inherits the catalog value and only an explicit false clears a catalog default for an engine started with its own chat template. An unlisted model is forwarded as sent. When the key resolves, every system entry of a /v1/chat/completions payload for that model is merged, in order and joined with a blank line, into one leading system message, and the other entries keep their relative order; a payload with no system entry, or exactly one that already leads, is not touched, a part the router cannot flatten, such as an image, is kept with the merged content becoming a parts array, and an entry whose content is neither a string nor a parts array leaves the request as sent. The merge runs at every /v1/chat/completions send site that carries a client conversation, on the OpenAI, Anthropic Messages (including the web_search query-extraction rounds), Responses, and ACP ingresses and on every fallback hop, resolved for the hop's own backend and model, so the mid-stream relay's next hop starts from the original array; it reads the published configuration and metadata on every dispatch, so a hot reload, an Admin backend edit, a config-sync change, or a metadata drop-in applies to the next request; and it is inert on anthropic, gemini, and bedrock backend types, whose dispatch does not send the array through these sites. It is a property of the model rather than the default because the merge keeps the instruction's priority but not its position, so one meant to apply from turn N on applies from the start; a configuration and catalog without the key dispatch exactly as before.

• Emit a usage chunk on native-shape Gemini chat streams (#1671). transform_native_gemini_event never read usageMetadata, and an event carrying only usageMetadata never reached it at all, because the stream dispatch routed only candidates events to the native path and an event without choices was forwarded verbatim. A client streaming from a Gemini-compatible backend that answers /v1/chat/completions in the native candidates shape therefore received a raw Gemini object, and the router recorded the request with zero prompt and completion tokens, so the cached-input split #1659 added had nothing to act on there. The native path now appends one OpenAI usage chunk on the stream's terminal event, choices: [] beside a usage object built by the same converter the non-streaming transform uses, so a Gemini context-cache hit arrives as prompt_tokens_details.cached_tokens and streaming and non-streaming report identical counts. The chunk is emitted once per stream rather than once per event carrying usageMetadata: Gemini repeats cumulative totals on every native event, and the tap that writes the usage record keeps the first chunk that parses as usage, so a chunk per event would have metered the first event's partial completion count as the whole completion. A stream whose events carry no usageMetadata is unchanged.

• Key the rate_limiting.limits.per_model bucket on a model name the router derives from the request instead of on the client-supplied X-Model header, and apply the dimension to inference requests for the first time (#1670). The limiter read X-Model ahead of everything else and the value it returned chose the bucket. The header was undocumented, nothing in the router set it, and no trust boundary filtered it, so any caller (an anonymous one, in permissive auth mode) could drain a bucket the whole fleet shares for a model they were not calling (GET /v1/models costs 0.1, so ten requests carrying a victim's name emptied a token of that model's bucket), name an unlimited model to skip a configured limit outright, or read a shared bucket's level out of the refusal's x-ratelimit-* headers. At the same time the dimension never saw a real inference request's model, because nothing read the request body: a chat completion was charged only when the client volunteered the header. A new middleware, mounted inner to authentication and outer to the limiter, now resolves the name from the request path on GET /v1/models/{model} and from the top-level model member of the JSON body on /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/rerank, /embed_sparse, /v1/responses, /v1/responses/compact, /v1/images/generations, and /anthropic/v1/messages, reading the body once within the 10 MB limit those routes already carry and handing it on unchanged. The bucket keys on the client-facing name exactly as the request spelled it, matched exactly, so a limit on a model experiment's client-facing model bounds the experiment as a whole rather than either variant's upstream capacity, and an upstream model reachable under two client-facing names through model_aliases needs an entry for each name. The multipart image edit and variation routes and /anthropic/v1/messages/count_tokens stay uncovered and are documented as such. A body with no model, a body that is not JSON or is malformed, and a body past the peek cap each leave the request untouched for the handler to validate and reject in its own error shape, and each is counted on rate_limit_skipped_total{dimension="model"}, which this dimension previously left silent. An operator who was relying on X-Model to select a bucket must send the model in the request body or path instead.

• Stop writing a prefix of the caller's raw credential into the model-listing rate-limit logs (#1669). extract_client_id derived the bucket key from the first 16 characters after Bearer, and both list handlers interpolated that value into the rejection warn!, so at the shipped logging.level: info a default deployment recorded a stable, vendor-marked prefix of every caller's key, and a key shorter than 16 characters went in whole. The bucket key is now derived from the API key id. No route in this repository mounts the two handlers, so the leak was latent in a public library handler reachable by an embed consumer or a future route edit rather than active.

• Let --no-default-features --features redis-cache compile again (PR #1623). src/infrastructure/cache/redis_store.rs records operation latency through RouterMetrics::global().kv_cache.record_redis_latency, and the no-op KvCacheMetrics stand-in compiled without metrics had no such method, so that graph failed with three E0599 errors. No shipped graph builds redis-cache without metrics, which is why nothing caught it.