Realtime WebSocket Proxy¶

The router can serve OpenAI Realtime-compatible speech backends over GET /v1/realtime?model=<id>, with GET /realtime as a compatibility alias for SDKs (OpenAI SDK, Pipecat) that use the short path. After the WebSocket upgrade, the router relays text and binary frames byte-identical in both directions: there is no protocol adaptation, so any backend that speaks the OpenAI Realtime event shapes (session.update, input_audio_buffer.append, response.created, response.done, function-call events) works unmodified.

The endpoint is compiled under the realtime Cargo feature, which is part of the default feature set, and activates only when the realtime config section is present with enabled: true. Absent, disabled, or built without the feature, the endpoint answers 404.

For a backend URL that maps to wss, the router verifies the backend certificate against the operating system trust store. Install a private backend CA in the host or container trust store; untrusted certificates and host-name mismatches are rejected.

Configuration¶

realtime:
  enabled: true
  max_sessions: 256
  handshake_timeout: "10s"
  idle_timeout: "60s"
  max_session_duration: "0"
  backend_path: "/v1/realtime"

Setting idle_timeout: "0" while max_session_duration keeps its "0" default leaves a session with no deadline at all. A half-open client (one whose host died without sending a TCP FIN) then pins one of the max_sessions permits and its backend session until the operating system's TCP keepalive expires, which is commonly two hours or disabled outright. The router logs a startup warning for this combination, and max_sessions such clients will exhaust all permits for the endpoint.

Reload class: restart. The section is snapshotted when routes are built, so editing it (including max_sessions) requires a process restart.

The HTTP timeouts section never applies to a realtime session. A full-duplex speech session has no first-byte or total HTTP deadline, so request.streaming.first_byte (enforced on SSE streams), request.standard.total, and timeouts.limits do not affect this path. The timers above are the only session deadlines.

Request flow¶

1. The upgrade request passes through the standard API auth middleware, so in api_keys.mode: blocking deployments the client authenticates with Authorization: Bearer <key> exactly like an HTTP request.
2. The Origin header is checked (see Browser origins below). A browser page whose origin is neither same-origin with the router nor allowed by the server.cors section is rejected 403 before anything else happens. Clients that send no Origin are unaffected.
3. The model query parameter is required (400 when missing) and resolves through the shared metadata matching pipeline, including aliases and quantization/date suffix normalization. An unknown model is rejected 404; a model whose metadata lacks the audio capability is rejected 400. Before any of that resolution, the requested name goes through the key's allowed_models gate, the same one every HTTP path uses: a name outside a non-empty list is rejected 403 with a permission_error body, before the handshake completes and before any backend is dialed. The gate judges the name the client sent, so an alias of a model the key may not use is refused too.
4. Backend candidates come from the same model lookup and routing filters as /v1/chat/completions, and the session backend is chosen through the shared selection and circuit-admission path:
   • When no backend is admissible (all unhealthy or circuits open), the upgrade is rejected 503.
   • When the client's raw model string matches no backend directly but its resolved canonical catalog id does (an alias lookup), the router dispatches that canonical id to the backend in the ?model= query parameter instead of the alias. The backend is always asked for a model it actually serves, while any client-facing 404 still echoes the string the caller sent.
5. The authenticated key's allowed_backends allow-list narrows those candidates, just as on every HTTP path:
   • A key whose allow-list names no backend serving the model is rejected 403 with a permission_error body. This is distinct from the 503 above because "this key may not use this model" and "every backend for this model is down right now" are different conditions, and only the latter is worth retrying.
   • An empty or absent list, or an anonymous caller under api_keys.mode: permissive, is unrestricted.
6. An x-backend header on the upgrade request names a preferred backend among the surviving candidates:
   • It is honored when the named backend is visible, allow-listed, among the candidates, and admissible, and silently ignored otherwise, so an unusable value never fails the handshake.
   • Because a WebSocket session selects a backend once, an honored preference binds the backend for the entire session lifetime with no later fallback. The 101 upgrade response does not include an x-served-backend header.
7. The router dials the backend's realtime endpoint, authenticating with the backend's configured credentials (api_key, or the hub-delivered credential in control-plane deployments). The client's Authorization header is never forwarded to the backend.
8. Frames relay in both directions until one side closes, a timer fires, or a leg fails. Close frames propagate with their originating code and reason; if one leg terminates abnormally, the other is closed with close code 1011. Backend dial failures and session establishment feed the same circuit breaker as HTTP traffic, so WebSocket and HTTP dispatch share failover state per backend.

Frame payloads carry raw user speech and are never logged at any level. Logs and metrics record event kinds, counts, sizes, durations, and outcomes only.

Browser origins¶

A WebSocket handshake is exempt from the same-origin policy and from CORS, and the page that opens it gets full read access to the stream. Without a check, any web page an operator visits could open a realtime session against a router it can reach (localhost, an intranet host), stream audio to the speech backend, and read the model's replies. The router therefore inspects Origin on the upgrade request:

To let a browser front-end on a different origin open realtime sessions, name it in the CORS section:

server:
  cors:
    enabled: true
    allow_origins:
      - "https://console.example.com"
      - "http://localhost:*"

The literal null origin is refused even under a * pattern: it is the opaque origin a sandboxed iframe or a data: document presents, so it identifies nothing.

Usage metering¶

The relay meters backend-reported usage without modifying the relayed frames. After forwarding a backend-to-client text frame, the router inspects a copy for two events: response.created (when a response starts) and response.done (the final usage and status). Each response.done emits one usage record through the same recorder used for HTTP completions, feeding /admin/stats, the per-key llm_tokens_total counter, and (when control_plane.enabled: true) the hub usage pipeline.

Each record carries:

The outcome follows response.status:

A failed or cancelled response is recorded even when it carries no usage, with zero tokens and token_source: none, because it still reached a backend.

Limits:

• A text frame above 256 KiB is relayed unchanged and never parsed (counted as oversized), and a text frame that is not a readable event is counted as unparsed. Neither changes the session.
• A response interrupted before response.done (when a backend drops or a client disconnects) is counted as session_closed_in_flight and records nothing. Because the protocol reports no usage before response.done, partial generation is not observable from the frames.
• Binary frames and client-to-backend frames are never parsed.
• Up to 16 responses per session are tracked between response.created and response.done; past that the oldest is dropped, and its later response.done records latency_ms: 0.
• A response.done whose frame the client sink had not yet accepted when the 5 s close grace period dropped the relay is observed too late to record: it is counted as session_closed_in_flight along with the other unfinished responses.
• Frame contents are never logged (no transcript, response id, or frame model string). A debug line per record logs only the backend, outcome, attempt number, token counts, the router's request_id, and the fixed status_reason set (max_output_tokens, content_filter, turn_detected, client_cancelled, other, absent).

Example: NVIDIA NemotronLabs VoiceChat¶

The NemotronLabs VoiceChat 11B container serves inference exclusively over an OpenAI Realtime-compatible WebSocket on /v1/realtime. Point a backend at the container, list the model, and enable the section:

backends:
  - name: voicechat
    url: "http://10.0.0.5:8000"
    type: vllm
    models:
      - nvidia-nemotronlabs-voicechat-11b

realtime:
  enabled: true
  idle_timeout: "120s"

Clients then connect with any OpenAI Realtime-compatible SDK:

ws://router:8080/v1/realtime?model=nvidia-nemotronlabs-voicechat-11b

Audio rides inside input_audio_buffer.append events as base64-encoded PCM16 (24 kHz mono little-endian toward the model; the container resamples internally), and the server emits standard response.* events, including response.function_call_arguments.done for tool calls. The router relays all of it untouched.

Health checks remain plain HTTP probes (health_check.endpoint per backend), which works for WS-only inference containers that still expose an HTTP health endpoint.

Metrics¶

With the metrics feature enabled, the proxy exports:

Limits and non-goals¶

• Message and frame sizes are capped at 4 MiB on both legs.
• Size max_sessions against available memory. The 4 MiB message ceiling applies per leg, so worst-case transient memory is roughly max_sessions * 2 * 4 MiB, about 2 GiB at the default max_sessions: 256. That is the ceiling a session can reach while buffering one oversized message per leg, not the steady state: ordinary realtime traffic is small frames, and steady-state read/write buffering is roughly 256 KiB per session.
• Browser-style subprotocol authentication (openai-insecure-api-key.<key>) is not supported; clients must send a standard Authorization header when auth is enabled.
• The router does not adapt non-OpenAI-Realtime speech protocols or transport WebRTC/SIP. Usage is read from response.done only (see Usage metering); no other event is metered.