Realtime WebSocket Proxy¶

The router can serve OpenAI Realtime-compatible speech backends over GET /v1/realtime?model=<id>, with GET /realtime as a compatibility alias for SDKs (OpenAI SDK, Pipecat) that use the short path. After the WebSocket upgrade, the router relays text and binary frames byte-identical in both directions: there is no protocol adaptation, so any backend that speaks the OpenAI Realtime event shapes (session.update, input_audio_buffer.append, response.created, response.done, function-call events) works unmodified.

The endpoint is compiled under the realtime Cargo feature, which is part of the default feature set, and activates only when the realtime config section is present with enabled: true. Absent, disabled, or built without the feature, the endpoint answers 404.

Configuration¶

realtime:
  enabled: true
  max_sessions: 256
  handshake_timeout: "10s"
  idle_timeout: "60s"
  max_session_duration: "0"
  backend_path: "/v1/realtime"

Setting idle_timeout: "0" while max_session_duration keeps its "0" default leaves a session with no deadline at all, so a half-open client (one whose machine died without sending a TCP FIN) pins one of the max_sessions permits and its backend session until the operating system's TCP keepalive expires, which is commonly two hours or disabled outright; the router logs a startup warning for this combination, and max_sessions such clients disable the endpoint.

Reload class: restart. The section is snapshotted when routes are built, so editing it (including max_sessions) requires a process restart.

The HTTP timeouts section never applies to a realtime session: a full-duplex speech session has no first byte or total deadline, so neither request.streaming.first_byte (which is enforced on SSE streams) nor request.standard.total reaches this path, and neither does timeouts.limits. The timers above are the only session deadlines.

Request flow¶

1. The upgrade request passes through the standard API auth middleware, so in api_keys.mode: blocking deployments the client authenticates with Authorization: Bearer <key> exactly like an HTTP request.
2. The Origin header is checked (see Browser origins below). A browser page whose origin is neither same-origin with the router nor allowed by the server.cors section is rejected 403 before anything else happens. Clients that send no Origin are unaffected.
3. The model query parameter is required (400 when missing) and resolves through the shared metadata matching pipeline, including aliases and quantization/date suffix normalization. An unknown model is rejected 404; a model whose metadata lacks the audio capability is rejected 400.
4. Backend candidates come from the same model lookup and routing filters as /v1/chat/completions, and the session backend is chosen through the shared selection and circuit-admission seam. When no backend is admissible (all unhealthy or circuits open), the upgrade is rejected 503. When the client's raw model string matches no backend directly but its resolved canonical catalog id does (an alias lookup), the router dispatches that canonical id to the backend in the ?model= query parameter instead of the alias, so the backend is always asked for a model it actually serves; the client-facing 404 still echoes the string the caller sent.
5. The authenticated key's allowed_backends allow-list narrows those candidates, exactly as it does on every HTTP path. A key whose allow-list names no backend serving the model is rejected 403 with a permission_error body, which is deliberately distinct from the 503 above: "this key may not use this model" and "every backend for this model is down right now" are different facts, and only one is worth retrying. An empty or absent list, and an anonymous caller under api_keys.mode: permissive, are unrestricted.
6. An x-backend header on the upgrade request names a preferred backend among the surviving candidates. It follows the same rules as on every other path: honored when the named backend is visible, allow-listed, among those candidates, and admissible, and silently ignored otherwise, so an unusable value never fails the handshake. What differs here is scope. The session takes exactly one selection, so an honored preference binds the backend for the session's whole lifetime rather than for one attempt, and there is no later selection to fall back to. The 101 upgrade response carries no x-served-backend header.
7. The router dials the backend's realtime endpoint, authenticating with the backend's configured credentials (api_key, or the hub-delivered credential in control-plane deployments). The client's Authorization header is never forwarded to the backend.
8. Frames relay in both directions until one side closes, a timer fires, or a leg fails. Close frames propagate with their originating code and reason; if one leg terminates abnormally, the other is closed with close code 1011. Backend dial failures and session establishment feed the same circuit breaker as HTTP traffic, so WebSocket and HTTP dispatch share failover state per backend.

Frame payloads carry raw user speech and are never logged at any level. Logs and metrics record event kinds, counts, sizes, durations, and outcomes only.

Browser origins¶

A WebSocket handshake is exempt from the same-origin policy and from CORS, and the page that opens it gets full read access to the stream. Without a check, any web page an operator visits could open a realtime session against a router it can reach (localhost, an intranet host), stream audio to the speech backend, and read the model's replies. The router therefore inspects Origin on the upgrade request:

To let a browser front-end on a different origin open realtime sessions, name it in the CORS section:

server:
  cors:
    enabled: true
    allow_origins:
      - "https://console.example.com"
      - "http://localhost:*"

The literal null origin is refused even under a * pattern: it is the opaque origin a sandboxed iframe or a data: document presents, so it identifies nothing.

Example: NVIDIA NemotronLabs VoiceChat¶

The NemotronLabs VoiceChat 11B container serves inference exclusively over an OpenAI Realtime-compatible WebSocket on /v1/realtime. Point a backend at the container, list the model, and enable the section:

backends:
  - name: voicechat
    url: "http://10.0.0.5:8000"
    type: vllm
    models:
      - nvidia-nemotronlabs-voicechat-11b

realtime:
  enabled: true
  idle_timeout: "120s"

Clients then connect with any OpenAI Realtime-compatible SDK:

ws://router:8080/v1/realtime?model=nvidia-nemotronlabs-voicechat-11b

Audio rides inside input_audio_buffer.append events as base64-encoded PCM16 (24 kHz mono little-endian toward the model; the container resamples internally), and the server emits standard response.* events, including response.function_call_arguments.done for tool calls. The router relays all of it untouched.

Health checks remain plain HTTP probes (health_check.endpoint per backend), which works for WS-only inference containers that still expose an HTTP health endpoint.

Metrics¶

With the metrics feature enabled, the proxy exports:

Limits and non-goals¶

• Message and frame sizes are capped at 4 MiB on both legs.
• Size max_sessions against available memory. The 4 MiB message ceiling applies per leg, so worst-case transient memory is roughly max_sessions * 2 * 4 MiB, about 2 GiB at the default max_sessions: 256. That is the ceiling a session can reach while buffering one oversized message per leg, not the steady state: ordinary realtime traffic is small frames, and steady-state read/write buffering is roughly 256 KiB per session.
• Browser-style subprotocol authentication (openai-insecure-api-key.<key>) is not supported; clients must send a standard Authorization header when auth is enabled.
• The router does not adapt non-OpenAI-Realtime speech protocols, extract token usage from response.done, or transport WebRTC/SIP.