Skip to content

API Reference

Continuum Router exposes an OpenAI-compatible API with additional administrative endpoints for monitoring and management. This document covers all available endpoints, request/response formats, and error handling.

Table of Contents

Overview

Base URL

http://localhost:8080

Content Type

All requests and responses use application/json unless otherwise specified.

OpenAI Compatibility

Continuum Router provides OpenAI-compatible API surfaces for: - Chat completions with streaming - Text completions - Embeddings - Reranking (Cohere-compatible /v1/rerank) - Sparse embeddings (TEI/Jina-compatible /embed_sparse) - Image generation - Image editing/inpainting - Image variations - Files API (upload, list, retrieve, delete) - File resolution in chat completions (image_file references) - Model listing - Error response formats

Authentication

Continuum Router supports API key authentication with configurable enforcement modes.

Authentication Modes

The router supports two authentication modes for API endpoints:

Mode Behavior
permissive (default) Requests without API key are allowed. Requests with valid API keys are authenticated and can access user-specific features.
blocking Only authenticated requests are processed. Requests without valid API key receive 401 Unauthorized.

Configuration

api_keys:
  # Authentication mode: "permissive" (default) or "blocking"
  mode: blocking

  # API key definitions
  api_keys:
    - key: "${API_KEY_1}"
      id: "key-production-1"
      user_id: "user-admin"
      organization_id: "org-main"
      scopes: [read, write, files, admin]

Protected Endpoints (when mode is blocking)

API-key middleware protects all routes in the core API router: /v1/models*, /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/responses*, /v1/images/*, /v1/rerank, /embed_sparse, /anthropic/v1/messages*, /anthropic/v1/models, and feature-gated /v1/batches* routes.

Note: Health and version endpoints (/health, /version) are always accessible without authentication. Admin, Files, and Metrics endpoints have separate authentication mechanisms.

Making Authenticated Requests

Every protected route accepts the API key in either of two header forms: Authorization: Bearer <api_key> or the Anthropic-native x-api-key: <api_key>. Both forms work on every route, not only the /anthropic/* endpoints, so Anthropic SDKs and Claude Code (which set only ANTHROPIC_API_KEY and send it via x-api-key) authenticate the same way OpenAI-style clients do. When a request carries both headers, Authorization: Bearer takes precedence.

POST /v1/chat/completions HTTP/1.1
Authorization: Bearer sk-your-api-key
Content-Type: application/json

{
  "model": "gpt-4",
  "messages": [{"role": "user", "content": "Hello"}]
}
POST /v1/chat/completions HTTP/1.1
x-api-key: sk-your-api-key
Content-Type: application/json

{
  "model": "gpt-4",
  "messages": [{"role": "user", "content": "Hello"}]
}

Authentication Errors

When authentication fails, the API returns:

{
  "error": {
    "message": "Missing or invalid API key. Expected Authorization: Bearer <api_key> or x-api-key",
    "type": "authentication_error",
    "code": "invalid_api_key"
  }
}

Status Codes:

  • 401 Unauthorized: Missing or invalid API key

Core API Endpoints

Health Check

Check the health status of the router service.

GET /health

Response:

{
  "status": "ok",
  "service": "continuum-router",
  "version": "1.14.0"
}

Status Codes:

  • 200: Service is healthy

Version

Return the running router version.

GET /version

Response:

{
  "version": "1.14.0"
}

Status Codes:

  • 200: Always returned; no authentication required

List Models

Retrieve all available models from all healthy backends.

GET /v1/models

Response:

{
  "object": "list",
  "data": [
    {
      "id": "gpt-4",
      "object": "model",
      "created": 1677610602,
      "owned_by": "openai-compatible",
      "permission": [],
      "root": "gpt-4",
      "parent": null
    },
    {
      "id": "llama2:7b",
      "object": "model", 
      "created": 1677610602,
      "owned_by": "local-ollama",
      "permission": [],
      "root": "llama2:7b",
      "parent": null
    }
  ]
}

Status Codes:

  • 200: Models retrieved successfully
  • 503: All backends are unhealthy

Features:

  • Model Aggregation: Combines models from all healthy backends
  • Deduplication: Removes duplicate models across backends
  • Caching: An internal 60-second hard TTL with stale-while-revalidate and request coalescing
  • Health Awareness: Only includes models from healthy backends

Proxy a Backend Model List

Return the raw /v1/models response from one selected eligible backend instead of aggregating all backends. This compatibility endpoint uses normal backend selection, retry, health, and caller backend-visibility rules.

GET /v1/models/proxy

Use GET /v1/models for the router's deduplicated aggregate catalog. Use this endpoint only when a client needs one backend's provider-native model-list payload.

Status Codes:

  • 200: Selected backend response returned
  • 502: Backend connection or gateway failure
  • 503: No eligible healthy backend

Force-Refresh Models

Immediately invalidate the model-aggregation cache and synchronously re-aggregate from all configured backends. Returns the same OpenAI-compatible model list shape as GET /v1/models so the caller can use the response immediately.

This endpoint is designed for interactive "Refresh" actions in desktop clients (e.g. the "Refresh models" button in backend.ai-go). It bypasses the stale-while-revalidate and soft-TTL windows, triggering an upstream fetch from every backend before responding.

POST /v1/models/refresh

Response:

{
  "object": "list",
  "data": [
    {
      "id": "gpt-4o",
      "object": "model",
      "created": 1677610602,
      "owned_by": "openai-compatible"
    }
  ]
}

Status Codes:

  • 200: Cache cleared and fresh model list returned
  • 429 Too Many Requests: Force-refresh rate limit exceeded (3 requests per 5-second burst window, 12 per minute)

Rate Limiting:

The endpoint applies a tighter budget than the regular GET /v1/models because each call triggers an upstream fetch from every configured backend. The default limits are 3 requests per 5-second window and 12 per minute. Valid API keys receive separate buckets keyed by API key ID. Anonymous callers and requests with invalid bearer tokens share one global anonymous bucket, so spoofed Authorization, X-Forwarded-For, or X-Real-IP headers cannot bypass the force-refresh budget. Clients that exceed the limit receive 429 and should wait before retrying.

Cache policy:

The model-list cache currently uses internal fixed defaults: a 60-second hard TTL, an 80% soft TTL, stale-while-revalidate, request coalescing, and bounded empty-result backoff. These values and force-refresh availability are not fields in the public configuration schema.


Get Single Model

Retrieve information about a specific model, including its availability status and optional rich metadata.

GET /v1/models/{model}

Path Parameters:

Parameter Type Required Description
model string Yes Model identifier (e.g., "gpt-4", "llama2:7b")

Response (Basic):

{
  "id": "gpt-4",
  "object": "model",
  "created": 1677610602,
  "owned_by": "openai",
  "available": true
}

Response (With Extended Metadata):

When model metadata is available in model-metadata.yaml, the response includes additional fields:

{
  "id": "gpt-4o",
  "object": "model",
  "created": 1704067200,
  "owned_by": "openai",
  "available": true,
  "supported_methods": ["chat.completions"],
  "features": ["chat", "vision", "audio", "code"],
  "max_tokens": 16384,
  "metadata": {
    "display_name": "GPT-4o",
    "developer": "OpenAI",
    "summary": "Multimodal optimized model with text, image, and audio capabilities.",
    "knowledge_cutoff": "2023-10",
    "relative_speed": 4,
    "pricing": {
      "input_tokens": 2.50,
      "output_tokens": 10.0
    },
    "limits": {
      "context_window": 128000,
      "max_output": 16384
    }
  }
}

Response Fields:

Field Type Description
id string Model identifier
object string Object type (always "model")
created integer Unix timestamp when model was created
owned_by string Organization that owns/provides the model
available boolean Whether the model can currently be used (true if at least one healthy backend provides it)
supported_methods array (Optional) API methods this model supports (e.g., ["chat.completions"], ["images.generations"])
features array (Optional) Model capabilities (e.g., ["chat", "vision", "function_calling"])
max_tokens integer (Optional) Maximum output tokens for this model
metadata object (Optional) Rich metadata object with detailed model information

Metadata Object Fields:

Field Type Description
display_name string Human-readable display name for the model
developer string Developer or organization that created the model
summary string Brief summary describing the model's capabilities
knowledge_cutoff string Knowledge cutoff date (e.g., "2025-01")
relative_speed integer Relative speed indicator (1-5, where 1 is slowest and 5 is fastest)
pricing object Pricing information (input_tokens, output_tokens per 1K tokens)
limits object Model limits (context_window, max_output in tokens)

Supported Methods Mapping:

The supported_methods field is derived from model capabilities:

Capability API Method
chat, vision, code, reasoning, audio, video, function_calling, tool chat.completions
embedding embeddings
image_generation images.generations
image_edit images.edits
image_variation images.variations
moderation moderations

Status Codes:

  • 200: Model found and information returned
  • 404: Model does not exist in any configured backend

Features:

  • OpenAI-Compatible: Response format matches OpenAI API with additional extension fields
  • Health-Aware: The available field reflects real-time backend health status
  • Privacy: Does not expose internal backend information
  • Optional Metadata: Extended fields are present only when metadata is available
  • Rich Metadata: Includes display name, developer, capabilities, and context-window limits to inform client-side model selection

Model Availability Algorithm:

The available field is determined by the following algorithm:

Condition Result
Health checker enabled + At least one backend providing the model is healthy true
Health checker enabled + All backends are unhealthy false
Health checker disabled + Model's backend list is non-empty true
Health checker disabled + Model's backend list is empty false

The algorithm uses short-circuit evaluation: it returns true as soon as the first healthy backend is found, avoiding unnecessary health checks for remaining backends. This optimizes performance when backends are generally healthy.

Performance Optimization:

  • Fast Path: If the model cache is valid, lookup is O(n) where n = number of models
  • Slow Path: If cache is empty/expired, triggers one-time aggregation with singleflight protection to prevent cache stampede
  • Stale-While-Revalidate: Cache serves stale data while refreshing in background

Example Request:

curl http://localhost:8080/v1/models/gpt-4o

Example Response (Model Available with Full Metadata):

{
  "id": "gpt-4o",
  "object": "model",
  "created": 1704067200,
  "owned_by": "openai",
  "available": true,
  "supported_methods": ["chat.completions"],
  "features": ["chat", "vision", "audio", "code"],
  "max_tokens": 16384,
  "metadata": {
    "display_name": "GPT-4o",
    "developer": "OpenAI",
    "summary": "Multimodal optimized model with text, image, and audio capabilities; faster and cheaper than GPT-4 Turbo.",
    "knowledge_cutoff": "2023-10",
    "relative_speed": 4,
    "pricing": {
      "input_tokens": 2.50,
      "output_tokens": 10.0
    },
    "limits": {
      "context_window": 128000,
      "max_output": 16384
    }
  }
}

Example Response (Model Without Extended Metadata):

For models without metadata in model-metadata.yaml, only the basic fields are returned:

{
  "id": "custom-model",
  "object": "model",
  "created": 1677610602,
  "owned_by": "local",
  "available": true
}

Example Response (Model Exists but Unavailable):

{
  "id": "gpt-4o",
  "object": "model",
  "created": 1704067200,
  "owned_by": "openai",
  "available": false,
  "supported_methods": ["chat.completions"],
  "features": ["chat", "vision", "audio", "code"],
  "max_tokens": 16384,
  "metadata": {
    "display_name": "GPT-4o",
    "developer": "OpenAI"
  }
}

Chat Completions

Generate chat completions using the OpenAI Chat API format.

POST /v1/chat/completions

Request Body:

{
  "model": "gpt-3.5-turbo",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user", 
      "content": "Explain quantum computing in simple terms."
    }
  ],
  "temperature": 0.7,
  "max_tokens": 150,
  "top_p": 1.0,
  "frequency_penalty": 0.0,
  "presence_penalty": 0.0,
  "stream": false,
  "stop": null,
  "logit_bias": {},
  "user": "user123"
}

Parameters:

Parameter Type Required Description
model string Yes Model identifier (must be available on at least one healthy backend)
messages array Yes Array of message objects with role and content
temperature number No Sampling temperature (0.0 to 2.0, default: 1.0)
max_tokens integer No Maximum tokens to generate
top_p number No Nucleus sampling parameter (0.0 to 1.0)
frequency_penalty number No Frequency penalty (-2.0 to 2.0)
presence_penalty number No Presence penalty (-2.0 to 2.0)
stream boolean No Enable streaming response (default: false)
stop string/array No Stop sequences
logit_bias object No Token logit bias
user string No User identifier for tracking
reasoning_effort string No Reasoning effort level for reasoning-capable models. Valid values vary by backend and model (see below). OpenAI values are validated exactly and are never silently downgraded. auto is reserved for adaptive thinking on Anthropic models.
reasoning object No Alternative nested format: {"effort": "high"}. Automatically normalized to reasoning_effort.

Valid reasoning_effort values by backend:

Backend Effort Levels Token Conversion Notes
OpenAI O-series (o1, o3, o4-mini) low, medium, high Native Exact model validation
OpenAI GPT-5 (including mini/nano) minimal, low, medium, high Native Original GPT-5 family only
OpenAI GPT-5.1 none, low, medium, high Native minimal is not accepted
OpenAI GPT-5.2 through GPT-5.5 none, low, medium, high, xhigh Native Exact model validation
OpenAI GPT-5.6 none, low, medium, high, xhigh, max Native max is GPT-5.6-only
OpenAI GPT-5 Pro high Native via Responses bridge Responses API only
OpenAI GPT-5.⅖.⅘.5 Pro medium, high, xhigh Native via Responses bridge Responses API only
Anthropic Claude (Opus 4.7, Opus 4.6, Sonnet 4.6, etc.) none, minimal, auto, low, medium, high, xhigh* budget_tokens or adaptive + output_config.effort *xhigh maps to max on Opus 4.6/4.7 and downgrades to high elsewhere; Opus 4.7 drops temperature/top_p/top_k
Gemini (3.x, 2.5-pro, 2.x-flash) none, auto*, minimal, low, medium, high Native none only for Flash; *auto downgraded to medium
Generic/llama.cpp Any Pass-through Backend handles validation

For detailed conversion tables and backend-specific behavior, see Reasoning Effort Architecture.

For OpenAI models, the router applies the same exact model-specific validation to streaming and non-streaming requests. Unsupported values return 400 invalid_request_error; the router never rewrites auto, xhigh, or max to a different effort. Other providers retain their provider-specific conversion behavior shown above.

Response (Non-streaming):

{
  "id": "chatcmpl-123456789",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "gpt-3.5-turbo",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Quantum computing uses quantum mechanical phenomena like superposition and entanglement to process information..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 150,
    "total_tokens": 175
  }
}

Response (Streaming): When stream: true, the response uses Server-Sent Events (SSE):

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-3.5-turbo","choices":[{"delta":{"role":"assistant","content":""},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-3.5-turbo","choices":[{"delta":{"content":"Quantum"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-3.5-turbo","choices":[{"delta":{"content":" computing"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-3.5-turbo","choices":[{"delta":{},"index":0,"finish_reason":"stop"}]}

data: [DONE]

Status Codes:

  • 200: Completion generated successfully
  • 400: Invalid request format or parameters
  • 404: Model not found on any healthy backend
  • 502: Backend connection error
  • 504: Request timeout
  • 503: All backends unhealthy

Features:

  • Model-Based Routing: Automatically routes to backends serving the requested model
  • Load Balancing: Distributes load across healthy backends
  • Streaming Support: Real-time response streaming via SSE
  • Error Recovery: Automatic retry on transient failures
  • Request Deduplication: Prevents duplicate processing of identical requests
  • Reasoning Parameter Normalization: Automatically normalizes nested reasoning format to flat reasoning_effort format; removes reasoning parameters for models that don't support them
  • Transparent Responses-API routing: When the requested model is marked responses_only: true (e.g., gpt-5-pro, gpt-5.2-pro, gpt-5.4-pro, gpt-5.5-pro), the router transparently dispatches to the upstream /v1/responses endpoint and translates the result back into a strict-mode chat.completion (or chat.completion.chunk for streaming) envelope, so clients keep using /v1/chat/completions unchanged. Only OpenAI and Azure OpenAI backends serve these models; pairing a responses_only model with another backend type produces a 400 invalid_request_error before dispatch. See Responses-API-only Models for the full list and configuration.

Responses API

Generate responses using OpenAI's Responses API format. This endpoint provides an alternative interface to Chat Completions, internally converting requests to the Chat Completions format for backend processing.

POST /v1/responses

Request Body:

{
  "model": "gpt-4o",
  "input": "Explain quantum computing in simple terms.",
  "instructions": "You are a helpful assistant that explains complex topics simply.",
  "max_output_tokens": 1000,
  "temperature": 0.7,
  "reasoning": {
    "effort": "high"
  }
}

Parameters:

Parameter Type Required Description
model string Yes Model identifier (must be available on at least one healthy backend)
input string/array Yes The input text or array of input items
instructions string No System instructions for the model (converted to system message)
max_output_tokens integer No Maximum tokens to generate
temperature number No Sampling temperature (0.0 to 2.0)
top_p number No Nucleus sampling parameter (0.0 to 1.0)
stream boolean No Enable streaming response (default: false)
store boolean No Whether to persist the response in session storage (default: true). Set to false for ephemeral requests that should not be stored (used by Codex)
include_reasoning boolean No Include reasoning content in the response
reasoning object No Reasoning configuration with nested format (see below)
tools array No List of tools available for the model (flat format)
tool_choice string/object No Controls tool usage
previous_response_id string No Reference to a previous response for multi-turn conversations

Tool Definition Format:

The Responses API uses a flat tool format where function properties are at the same level as the type field. This differs from the Chat Completions API which uses a nested function object.

{
  "type": "function",
  "name": "get_weather",
  "description": "Get weather for a location",
  "parameters": {
    "type": "object",
    "properties": {
      "location": {"type": "string", "description": "City name"}
    },
    "required": ["location"]
  },
  "strict": true
}
Field Type Required Description
type string Yes Must be "function"
name string Yes Name of the function
description string No Description of what the function does
parameters object No JSON Schema for function parameters
strict boolean No Enable strict parameter validation

Other supported tool types include code_interpreter, file_search, web_search, and computer_use.

Multi-Modal Input Types:

When input is an array of items, each item can be a message containing multi-modal content parts. The following content part types are supported:

Type Description
text Plain text content
input_text Text content for user messages (alternative format for Responses API)
output_text Text content for assistant messages; differentiates assistant-generated content from user input (input_text)
input_file File content (PDF, images, or other files)
input_image Image content
image_url Image from URL or base64 data

Message Roles:

Each message item carries one of five roles. The router accepts all five for /v1/responses and translates them per backend.

Role Purpose Per-backend mapping
developer Application-developer instructions for OpenAI reasoning models (GPT-5.x, o-series). Highest priority in the chain of command and replaces system for those models. OpenAI: passthrough as developer. Anthropic: merged into the system parameter (concatenated with any system text using \n\n). Gemini: merged into system_instruction. Ollama / vLLM / LocalAI / LM Studio: mapped to system.
system Legacy application instructions. Behaves as before.
user End-user input. Behaves as before.
assistant Prior model output. Behaves as before.
tool Tool-call result returned to the model. Behaves as before.

The router preserves arrival order when multiple developer and system messages are present. When the target backend (Anthropic, Gemini) only accepts a single instruction slot, the router concatenates them with a blank-line separator so no instruction text is silently dropped.

Item References:

An input item of the form {"type": "item_reference", "id": "item_..."} points at an output item from a previously stored response. This is how the OpenAI and Vercel AI SDKs submit multi-step tool round-trips by default: instead of echoing the full function-call item back, step 2 references the item produced in step 1.

  • On the OpenAI / Azure OpenAI passthrough path, references are forwarded unchanged and the upstream resolves them.
  • On the converted paths (Anthropic, Gemini, and chat-completions-backed routes), the router resolves each reference from its session store before dispatch, rewriting it to the inline function_call, message, or function_call_output item it points to. References are de-duplicated by call_id (first wins).
  • Streamed responses participate as well: the accumulated response is stored when the stream completes, so a follow-up request can reference items produced by a streaming step.
  • An unresolvable reference returns HTTP 400 naming the id, and a request may carry at most 256 references.

Input File Format:

The input_file content part supports three input methods:

{
  "type": "input_file",
  "filename": "document.pdf",
  "file_data": "data:application/pdf;base64,JVBERi0xLjQ..."
}
Field Type Description
filename string Optional filename for the file
file_data string Base64 data URL (e.g., data:application/pdf;base64,...)
file_url string External URL to the file (validated for SSRF)
file_id string Reference to a file uploaded via Files API

Input Image Format:

{
  "type": "input_image",
  "image_url": "data:image/png;base64,iVBORw0KGgo...",
  "detail": "high"
}
Field Type Description
image_url string Image URL or base64 data URL
file_id string Reference to an image file uploaded via Files API
detail string Image detail level: low, high, or auto (default)

Exactly one of image_url or file_id must be present. A part with neither is rejected with 400. When file_id is provided, the file is resolved to an inline base64 data URL before the request reaches the backend; file ownership is verified and a 10MB size limit applies, matching the input_file + file_id path.

Multi-Modal Request Example:

curl -X POST http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "input": [
      {
        "type": "message",
        "role": "user",
        "content": [
          {"type": "input_text", "text": "What does this document say?"},
          {"type": "input_file", "filename": "report.pdf", "file_data": "data:application/pdf;base64,..."}
        ]
      }
    ]
  }'

Security Notes:

  • External URLs in file_url are validated to prevent SSRF attacks
  • Private IP addresses and localhost URLs are rejected
  • Only HTTPS URLs are recommended for external files
  • The file_id field references files uploaded via the Files API. Files are resolved and converted to base64 before sending to backends. File ownership is verified and a 10MB size limit applies for file injection

Reasoning Parameter:

The reasoning parameter controls the reasoning effort level for reasoning-capable models. It uses a nested format:

{
  "reasoning": {
    "effort": "high"
  }
}

Valid effort values:

Value Description
none Disable reasoning on models that support it
minimal Minimal reasoning on the original GPT-5 family and supported non-OpenAI models
auto Adaptive thinking (Anthropic)
low Minimal reasoning effort, faster responses
medium Balanced reasoning effort
high Maximum standard reasoning effort
xhigh Extended reasoning on supported model families
max Maximum reasoning on GPT-5.6

The router automatically converts this nested format to the flat reasoning_effort format used by Chat Completions backends. Invalid effort levels are rejected with a 400 Bad Request error.

Example Request with Reasoning:

curl -X POST http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "o1",
    "input": "Solve this complex mathematical problem step by step.",
    "reasoning": {
      "effort": "high"
    }
  }'

Response:

{
  "id": "resp_abc123",
  "object": "response",
  "created_at": 1699000000,
  "model": "o1",
  "output": [
    {
      "type": "message",
      "id": "msg_001",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "Let me solve this step by step..."
        }
      ]
    }
  ],
  "usage": {
    "input_tokens": 25,
    "output_tokens": 150
  },
  "status": "completed"
}

Status Codes:

  • 200: Response generated successfully
  • 400: Invalid request format, parameters, or invalid reasoning effort value
  • 404: Model not found on any healthy backend
  • 502: Backend connection error
  • 504: Request timeout
  • 503: All backends unhealthy

Features:

  • Smart Routing: Routes requests based on backend capabilities - native pass-through for OpenAI/Azure, automatic conversion for others
  • Native Pass-through: For OpenAI and Azure OpenAI backends, requests are forwarded directly to /v1/responses endpoint, preserving all native features
  • Automatic Conversion: For other backends (Anthropic, Gemini, vLLM, Ollama, etc.), converts Responses API format to their native format
  • Reasoning Support: Full support for reasoning parameter with type-safe validation
  • Multi-Backend Support: Works with OpenAI, Anthropic, Gemini, Ollama, vLLM, and other backends
  • Streaming Support: Real-time response streaming via SSE when stream: true
  • Session Management: Supports multi-turn conversations via previous_response_id

Routing Strategy:

The router automatically determines the best strategy for each backend:

Backend Type Strategy Description
OpenAI Pass-through Native Responses API support - requests forwarded directly
Azure OpenAI Pass-through Native Responses API support - requests forwarded directly
Anthropic Native Convert Converted to native Anthropic Messages API format with full PDF/image support
Gemini Convert Converted to Gemini generateContent API format
vLLM Convert Converted to Chat Completions format
Ollama Convert Converted to Chat Completions format
LlamaCpp Convert Converted to Chat Completions format
Generic Convert Converted to Chat Completions format

Pass-through Benefits:

When using OpenAI or Azure OpenAI backends with pass-through mode:

  • Native PDF file support (Chat Completions only supports images)
  • Preserved reasoning state between turns for better performance
  • Access to built-in tools (web_search, file_search, etc.)
  • Better cache utilization (40-80% improvement per OpenAI documentation)
  • Full compatibility with the latest OpenAI Responses API features

Anthropic Native Conversion Benefits:

When using Anthropic (Claude) backends with native conversion:

  • Native PDF file support via Anthropic's document understanding
  • Image file support with automatic format detection
  • Extended thinking support for Claude 3+ models
  • SSRF protection for external file URLs
  • Media type whitelisting for security

Retrieve or Delete a Stored Response

Responses created with store: true can be retrieved or deleted by ID. Stored sessions are process-local and are scoped to the authenticated API-key principal (api_keys[].user_id) that created the response. Both Authorization: Bearer <key> and X-API-Key: <key> credentials use the same configured identity namespace. Missing, expired, unauthenticated, or differently scoped requests all return 404.

Migration from X-User-ID Scoping

X-User-ID is not an authorization boundary for the Responses API. The router ignores it for stored response access, previous_response_id context reconstruction, item-reference resolution, and local Files API ownership checks, and it is not forwarded to backends. Deployments that previously multiplexed multiple end users behind one shared router API key by setting X-User-ID must migrate to one router API key per isolated user or tenant, or place their own trusted service in front of the router and keep per-user state outside the router's stored Responses session and Files API ownership controls.

GET /v1/responses/{response_id}
DELETE /v1/responses/{response_id}

A successful GET returns the stored response object. A successful DELETE returns:

{
  "id": "resp_abc123",
  "object": "response.deleted",
  "deleted": true
}

Responses Compact

Compact a conversation context by summarizing or compressing a sequence of input items. For OpenAI and Azure OpenAI backends, the request is forwarded directly to the backend's native /v1/responses/compact endpoint. Other backends return HTTP 501 (Not Implemented), as context compaction is not natively supported.

POST /v1/responses/compact

Request Body:

{
  "model": "gpt-4o",
  "instructions": "Summarize the key points from this conversation.",
  "input": [
    {
      "type": "message",
      "role": "user",
      "content": [{"type": "input_text", "text": "What is the capital of France?"}]
    },
    {
      "type": "message",
      "role": "assistant",
      "content": [{"type": "output_text", "text": "The capital of France is Paris."}]
    }
  ]
}

Parameters:

Parameter Type Required Description
model string Yes Model identifier
input array Yes Array of input items representing the conversation to compact
instructions string No Optional instructions to guide the compaction process

Response:

{
  "id": "cmpt_abc123",
  "object": "response.compaction",
  "created_at": 1699000000,
  "output": [
    {
      "type": "text",
      "text": "Compacted conversation summary..."
    }
  ],
  "usage": {
    "input_tokens": 120,
    "output_tokens": 45
  }
}

Status Codes:

  • 200: Compaction successful
  • 400: Invalid request (empty model, empty input, instructions too long)
  • 501: Backend does not support native context compaction
  • 502: Backend connection error
  • 504: Request timeout

Batch API (control-plane builds)

The OpenAI-style Batch API is compiled only with the control-plane Cargo feature. It is absent from default source builds. Official release binaries compile the feature, but the routes remain inert unless control_plane.enabled, policy enforcement, and control_plane.batch.enabled are active and the Hub marks the presented key/tier as batch-eligible.

Method Path Purpose
POST /v1/batches Submit a provider batch
GET /v1/batches List batches owned by the calling key
GET /v1/batches/{batch_id} Retrieve owned batch status
POST /v1/batches/{batch_id}/cancel Cancel an owned batch
GET /v1/batches/{batch_id}/results Return completed results as JSONL

A hub-synced API key is required. Ownership checks deliberately return the same 404 for unknown and other-key IDs. Results return 409 until available. The selected batch backend may be OpenAI-compatible or native Anthropic; the router presents one common surface. x-continuum-batch: off always opts a request out and cannot be overridden by policy.


Image Generation

Generate images using OpenAI's DALL-E, GPT Image models, or Google's Nano Banana (Gemini) models.

POST /v1/images/generations

Request Body:

{
  "model": "dall-e-3",
  "prompt": "A serene Japanese garden with cherry blossoms",
  "n": 1,
  "size": "1024x1024",
  "quality": "standard",
  "response_format": "url"
}

Parameters:

Parameter Type Required Description
model string Yes Image model: dall-e-2, dall-e-3, gpt-image-1, gpt-image-1.5, gpt-image-1-mini, nano-banana, or nano-banana-pro
prompt string Yes Description of the image to generate
n integer No Number of images (1-10, varies by model)
size string No Image size (varies by model, see below)
quality string No Image quality (varies by model, see below)
style string No Image style: vivid or natural (DALL-E 3 only)
response_format string No Response format: url or b64_json
output_format string No Output file format: png, jpeg, webp (GPT Image models only, default: png)
output_compression integer No Compression level 0-100 for jpeg/webp (GPT Image models only)
background string No Background: transparent, opaque, auto (GPT Image models only)
stream boolean No Enable streaming for partial images (GPT Image models only, default: false)
partial_images integer No Number of partial images 0-3 during streaming (GPT Image models only)
user string No User identifier for tracking

Model-specific constraints:

Model Sizes n Quality Notes
dall-e-2 256x256, 512x512, 1024x1024 1-10 N/A Classic DALL-E 2
dall-e-3 1024x1024, 1792x1024, 1024x1792 1 standard, hd High quality with prompt revision
gpt-image-1 1024x1024, 1536x1024, 1024x1536, auto 1 low, medium, high, auto Latest GPT Image model, supports streaming
gpt-image-1.5 1024x1024, 1536x1024, 1024x1536, auto 1 low, medium, high, auto 4x faster, better text rendering
gpt-image-1-mini 1024x1024, 1536x1024, 1024x1536, auto 1 low, medium, high, auto Cost-effective option
nano-banana 256x256 to 1024x1024 1-4 N/A Gemini 2.5 Flash Image (fast)
nano-banana-pro 256x256 to 4096x4096 1-4 N/A Gemini 2.0 Flash Image (advanced, up to 4K)

Quality Parameter (GPT Image Models):

For GPT Image models, standard maps to medium and hd maps to high.

Quality Description
low Fast generation with lower quality
medium Balanced quality and speed (default)
high Best quality, slower generation
auto Model selects optimal quality

Output Format Options (GPT Image Models):

Format Description Supports Transparency
png Lossless format (default) Yes
jpeg Lossy format, smaller file size No
webp Modern format, good compression Yes

Note: Transparent background (background: "transparent") requires png or webp format.

Nano Banana (Gemini) Models:

Nano Banana provides access to Google's Gemini image generation capabilities through an OpenAI-compatible interface:

  • nano-banana: Maps to Gemini 2.5 Flash Image - fast, general-purpose image generation
  • nano-banana-pro: Maps to Gemini 2.0 Flash Image - advanced model with high-resolution support (up to 4K)

Nano Banana Size Mapping:

The router automatically converts OpenAI-style size parameters to Gemini's aspectRatio and imageSize format:

OpenAI Size Gemini aspectRatio Gemini imageSize Notes
256x256 1:1 1K Falls back to Gemini minimum
512x512 1:1 1K Falls back to Gemini minimum
1024x1024 1:1 1K Default
1536x1024 3:2 1K Landscape
1024x1536 2:3 1K Portrait
1024x1792 9:16 1K Tall portrait
1792x1024 16:9 1K Wide landscape
2048x2048 1:1 2K Pro only
4096x4096 1:1 4K Pro only
auto 1:1 1K Default fallback

The conversion sends the following Gemini API structure:

{
  "contents": [{"parts": [{"text": "Your prompt"}]}],
  "generationConfig": {
    "imageConfig": {
      "aspectRatio": "3:2",
      "imageSize": "1K"
    }
  }
}

Example Nano Banana Request:

{
  "model": "nano-banana",
  "prompt": "A white siamese cat with blue eyes, photorealistic",
  "n": 1,
  "size": "1024x1024",
  "response_format": "b64_json"
}

Response:

{
  "created": 1677652288,
  "data": [
    {
      "url": "https://oaidalleapiprodscus.blob.core.windows.net/...",
      "revised_prompt": "A tranquil Japanese garden featuring..."
    }
  ]
}

Response (with b64_json):

{
  "created": 1677652288,
  "data": [
    {
      "b64_json": "/9j/4AAQSkZJRgABAQAA...",
      "revised_prompt": "A tranquil Japanese garden featuring..."
    }
  ]
}

Nano Banana Response Notes:

  • When using response_format: "url" with Nano Banana, the image is returned as a data URL (data:image/png;base64,...) since Gemini's native API returns inline base64 data
  • The revised_prompt field contains any text response from Gemini describing the generated image

Streaming Image Generation (GPT Image Models):

When stream: true is specified for GPT Image models, the response will be streamed as Server-Sent Events (SSE):

Example Streaming Request:

{
  "model": "gpt-image-1",
  "prompt": "A beautiful sunset over mountains",
  "stream": true,
  "partial_images": 2,
  "response_format": "b64_json"
}

Streaming Response Format:

data: {"type":"image_generation.partial_image","partial_image_index":0,"b64_json":"...","created":1702345678}

data: {"type":"image_generation.partial_image","partial_image_index":1,"b64_json":"...","created":1702345679}

data: {"type":"image_generation.complete","b64_json":"...","created":1702345680}

data: {"type":"image_generation.usage","usage":{"input_tokens":25,"output_tokens":1024}}

data: {"type":"done"}

SSE Event Types:

Event Type Description
image_generation.partial_image Intermediate image during generation
image_generation.complete Final complete image
image_generation.usage Token usage information (for cost tracking)
done Stream completion marker

Example GPT Image Request with New Options:

{
  "model": "gpt-image-1.5",
  "prompt": "A white cat with blue eyes, photorealistic",
  "size": "auto",
  "quality": "high",
  "output_format": "webp",
  "output_compression": 85,
  "background": "transparent",
  "response_format": "b64_json"
}

Status Codes:

  • 200: Image(s) generated successfully
  • 400: Invalid request (e.g., invalid size for model, n > 1 for DALL-E 3)
  • 401: Invalid API key
  • 429: Rate limit exceeded
  • 500: Backend error
  • 503: Gemini backend unavailable (for Nano Banana models)

Timeout Configuration: Image generation requests use a configurable timeout (default: 3 minutes). See timeouts.request.image_generation in configuration.


Image Edit (Inpainting)

Edit existing images using OpenAI's inpainting capabilities. This endpoint allows you to modify specific regions of an image based on a text prompt and optional mask. Supports GPT Image models and DALL-E 2.

POST /v1/images/edits
Content-Type: multipart/form-data

Request Parameters (multipart/form-data):

Parameter Type Required Description
image file Yes The source image to edit (PNG, < 4MB, square)
prompt string Yes Description of the desired edit
mask file No Mask image indicating edit regions (PNG, same dimensions as image)
model string No Model to use (default: gpt-image-1)
n integer No Number of images to generate (1-10, default: 1)
size string No Output size (model-dependent, default: 1024x1024)
response_format string No Response format: url or b64_json (default: url)
user string No Unique user identifier for tracking

Supported Models and Sizes:

Model Sizes Notes
gpt-image-1 1024x1024, 1536x1024, 1024x1536, auto Latest GPT Image model (recommended)
gpt-image-1-mini 1024x1024, 1536x1024, 1024x1536, auto Cost-optimized version
gpt-image-1.5 1024x1024, 1536x1024, 1024x1536, auto Newest with improved instruction following
dall-e-2 256x256, 512x512, 1024x1024 Legacy DALL-E 2 model

Note: DALL-E 3 and Gemini (nano-banana) do NOT support image editing via this endpoint. Gemini uses semantic masking via natural language, which is incompatible with OpenAI's mask-based editing format.

Image Requirements:

  • Format: PNG only
  • Size: Less than 4MB
  • Dimensions: Must be square (width equals height)

Mask Requirements:

  • Format: PNG with alpha channel (RGBA)
  • Dimensions: Must match the source image exactly
  • Transparent areas: Indicate regions to edit/generate
  • Opaque areas: Indicate regions to preserve

Example Request:

curl -X POST http://localhost:8080/v1/images/edits \
  -F "image=@source_image.png" \
  -F "mask=@mask.png" \
  -F "prompt=A sunlit indoor lounge area with a pool containing a flamingo" \
  -F "n=1" \
  -F "size=1024x1024" \
  -F "response_format=url"

Example Request (without mask):

curl -X POST http://localhost:8080/v1/images/edits \
  -F "image=@source_image.png" \
  -F "prompt=Add a sunset in the background" \
  -F "n=1" \
  -F "size=512x512"

Response:

{
  "created": 1677652288,
  "data": [
    {
      "url": "https://oaidalleapiprodscus.blob.core.windows.net/..."
    }
  ]
}

Response (with b64_json):

{
  "created": 1677652288,
  "data": [
    {
      "b64_json": "/9j/4AAQSkZJRgABAQAA..."
    }
  ]
}

Status Codes:

  • 200: Image(s) edited successfully
  • 400: Invalid request (e.g., non-square image, invalid size, missing required field)
  • 401: Invalid API key
  • 503: OpenAI backend unavailable

Error Examples:

Non-square image:

{
  "error": {
    "message": "Image must be square (800x600 is not square)",
    "type": "invalid_request_error",
    "param": "image",
    "code": "image_not_square"
  }
}

Mask dimension mismatch:

{
  "error": {
    "message": "Mask dimensions (256x256) do not match image dimensions (512x512)",
    "type": "invalid_request_error",
    "param": "mask",
    "code": "dimension_mismatch"
  }
}

Unsupported model:

{
  "error": {
    "message": "Model 'dall-e-3' does not support image editing. Supported models: gpt-image-1, gpt-image-1-mini, gpt-image-1.5, dall-e-2. Note: dall-e-3 does NOT support image editing.",
    "type": "invalid_request_error",
    "param": "model",
    "code": "unsupported_model"
  }
}

Notes:

  • Supported models: gpt-image-1, gpt-image-1-mini, gpt-image-1.5, dall-e-2
  • DALL-E 3 does NOT support image editing via API
  • Gemini (nano-banana) is NOT supported - uses different editing approach (semantic masking)
  • When no mask is provided, the entire image may be modified
  • The source image should have transparent regions if editing without a mask
  • Request timeout uses the image generation timeout configuration

Image Variations

Generate variations of an existing image using OpenAI's DALL-E 2 model.

POST /v1/images/variations
Content-Type: multipart/form-data

Form Fields:

Parameter Type Required Description
image file Yes Source image for variations (PNG, < 4MB, must be square)
model string No Model to use (default: dall-e-2)
n integer No Number of variations to generate (1-10, default: 1)
size string No Output size: 256x256, 512x512, 1024x1024 (default: 1024x1024)
response_format string No Response format: url or b64_json (default: url)
user string No User identifier for tracking

Example Request:

curl -X POST http://localhost:8080/v1/images/variations \
  -F "image=@source_image.png" \
  -F "model=dall-e-2" \
  -F "n=2" \
  -F "size=512x512" \
  -F "response_format=url"

Response:

{
  "created": 1677652288,
  "data": [
    {
      "url": "https://oaidalleapiprodscus.blob.core.windows.net/..."
    },
    {
      "url": "https://oaidalleapiprodscus.blob.core.windows.net/..."
    }
  ]
}

Response (with b64_json):

{
  "created": 1677652288,
  "data": [
    {
      "b64_json": "/9j/4AAQSkZJRgABAQAA..."
    }
  ]
}

Model Support:

Model Variations Support Notes
dall-e-2 Yes (native) Full support, 1-10 variations
dall-e-3 No Not supported by OpenAI API
gpt-image-1 No Not supported
nano-banana No Gemini does not support variations API
nano-banana-pro No Gemini does not support variations API

Image Requirements:

  • Format: PNG only
  • Size: Less than 4MB
  • Dimensions: Must be square (width == height)
  • Supported input sizes: Any square dimensions (will be processed by the model)

Error Scenarios:

Error Status Description
Image not PNG 400 Only PNG format is supported
Image not square 400 Image dimensions must be equal
Image too large 400 Image exceeds 4MB size limit
Model not supported 400 Requested model doesn't support variations
Missing image 400 Image field is required
Invalid n value 400 n must be between 1 and 10
Invalid size 400 Size must be one of the supported values

Status Codes:

  • 200: Variation(s) generated successfully
  • 400: Invalid request (invalid format, non-square image, unsupported model)
  • 401: Invalid API key
  • 429: Rate limit exceeded
  • 500: Backend error
  • 503: Backend unavailable

Text Completions

Generate text completions using the OpenAI Completions API format.

POST /v1/completions

Request Body:

{
  "model": "gpt-3.5-turbo-instruct",
  "prompt": "Once upon a time in a distant galaxy",
  "max_tokens": 100,
  "temperature": 0.7,
  "top_p": 1.0,
  "frequency_penalty": 0.0,
  "presence_penalty": 0.0,
  "stream": false,
  "stop": null,
  "logit_bias": {},
  "user": "user123"
}

Response:

{
  "id": "cmpl-123456789",
  "object": "text_completion",
  "created": 1677652288,
  "model": "gpt-3.5-turbo-instruct",
  "choices": [
    {
      "text": ", there lived a young explorer named Zara who dreamed of discovering new worlds...",
      "index": 0,
      "finish_reason": "stop",
      "logprobs": null
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 90,
    "total_tokens": 100
  }
}

Status Codes: Same as Chat Completions


Embeddings

Generate embeddings for input text using the OpenAI Embeddings API format.

POST /v1/embeddings

Request Body:

{
  "model": "text-embedding-3-small",
  "input": "The quick brown fox jumps over the lazy dog",
  "encoding_format": "float",
  "dimensions": 512,
  "user": "user123"
}

Request Parameters:

Field Type Required Description
model string Yes ID of the model to use (e.g., text-embedding-3-small, gemini-embedding-2)
input string, array, or content parts Yes Input to embed. Can be a single string, array of strings, array of token arrays, or an array of multimodal content parts (for Gemini Embedding 2)
encoding_format string No Format to return the embeddings: float or base64 (default: float)
dimensions integer No Number of dimensions for the output embeddings (supported for text-embedding-3-* and gemini-embedding-2* models)
user string No Unique identifier for the end-user

Input Format Examples:

Single string:

{
  "model": "text-embedding-3-small",
  "input": "Hello, world!"
}

Array of strings:

{
  "model": "text-embedding-3-large",
  "input": [
    "First text to embed",
    "Second text to embed",
    "Third text to embed"
  ]
}

Token arrays:

{
  "model": "text-embedding-ada-002",
  "input": [[1234, 5678, 9012], [3456, 7890, 1234]]
}

Multimodal content parts (Gemini Embedding 2 only):

{
  "model": "gemini-embedding-2",
  "input": [
    {"type": "text", "text": "A photo of a golden retriever"},
    {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQ..."}}
  ]
}

Response:

{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [0.0023064255, -0.009327292, 0.0045318254, ...]
    }
  ],
  "model": "text-embedding-3-small",
  "usage": {
    "prompt_tokens": 8,
    "total_tokens": 8
  }
}

Response Fields:

Field Type Description
object string Always "list"
data array Array of embedding objects
data[].object string Always "embedding"
data[].index integer Index of the embedding in the input array
data[].embedding array Embedding vector (array of floats)
model string Model used to generate the embedding
usage object Token usage information
usage.prompt_tokens integer Number of tokens in the input
usage.total_tokens integer Total tokens used (same as prompt_tokens for embeddings)

Supported Models:

Backend Model Notes
OpenAI text-embedding-3-small 1536 dimensions by default, supports dimensions parameter
OpenAI text-embedding-3-large 3072 dimensions by default, supports dimensions parameter
OpenAI text-embedding-ada-002 1536 dimensions, legacy model
Gemini gemini-embedding-2 3072 dimensions, multimodal (text, image, audio, video, document)
Gemini gemini-embedding-2-preview Preview release of Gemini Embedding 2, multimodal
Gemini text-embedding-004 Text-only, via OpenAI-compatible endpoint
Self-hosted bge-m3 1024 dimensions, 100+ languages, 8192 context. Supports dense, sparse, and ColBERT retrieval
Self-hosted bge-large-en-v1.5 1024 dimensions, English-only, 512 context
Self-hosted multilingual-e5-large 1024 dimensions, 100+ languages, 514 context
vLLM Deployment-specific Depends on deployed model
llama.cpp Deployment-specific Native /v1/embeddings support
TEI Deployment-specific Hugging Face Text Embeddings Inference server
Ollama Deployment-specific Via Ollama embedding models

Status Codes:

  • 200: Embeddings generated successfully
  • 400: Invalid request (missing model/input, invalid dimensions)
  • 401: Invalid API key
  • 404: Model not found or doesn't support embeddings
  • 429: Rate limit exceeded
  • 500: Backend error
  • 503: Backend unavailable

Features:

  • Multiple Input Formats: Supports single string, array of strings, token arrays, or multimodal content parts
  • Multimodal Embeddings: gemini-embedding-2 and gemini-embedding-2-preview accept mixed text, image, audio, video, and document parts in a single request
  • Dimension Control: For text-embedding-3-* and gemini-embedding-2* models, specify custom dimensions for reduced vector size
  • Backend Agnostic: Routes to appropriate backend based on model
  • Load Balancing: Applies configured load balancing strategy
  • Error Handling: Returns 400 with a clear message when multimodal input is sent to a text-only model

Multimodal Content Part Types (Gemini Embedding 2 only):

Type Fields Constraints
text text (string) Must not be empty
image_url image_url.url (data URI or file://file-* reference) MIME: image/jpeg, image/png; max 20 MiB; up to 6 images per request
audio audio.data (base64), audio.format Formats: mp3, wav; max 20 MiB
video video.data (base64), video.format Formats: mp4, mov; max 20 MiB
document document.data (base64), document.format Formats: pdf; max 20 MiB

Example Request:

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "model": "text-embedding-3-small",
    "input": "The quick brown fox jumps over the lazy dog",
    "encoding_format": "float"
  }'

Example with Multiple Inputs:

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "model": "text-embedding-3-large",
    "input": [
      "First document text",
      "Second document text",
      "Third document text"
    ]
  }'

Example with Custom Dimensions:

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "model": "text-embedding-3-small",
    "input": "Sample text",
    "dimensions": 512
  }'

Example with Multimodal Input (Gemini Embedding 2):

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "model": "gemini-embedding-2",
    "input": [
      {"type": "text", "text": "A photo of a sunset over the ocean"},
      {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQ..."}}
    ],
    "dimensions": 768
  }'

Rerank

Rerank documents based on their relevance to a query using the Cohere-compatible Rerank API. This is commonly used as a second-stage retrieval step after initial vector search to improve accuracy.

POST /v1/rerank

Request Body:

{
  "model": "bge-reranker-v2-m3",
  "query": "What is machine learning?",
  "documents": ["Document 1 content", "Document 2 content", "Document 3 content"],
  "top_n": 3,
  "return_documents": false
}

Request Parameters:

Field Type Required Description
model string Yes ID of the reranking model to use (e.g., bge-reranker-v2-m3, rerank-english-v3.0, jina-reranker-v2-base-multilingual)
query string Yes The search query to compare documents against
documents array Yes List of documents to rerank. Can be an array of strings or an array of objects with a text field
top_n integer No Number of top results to return. If not specified, returns all documents ranked
return_documents boolean No Whether to return the document text in the response (default: false)
max_chunks_per_doc integer No Maximum number of chunks to process per document for long document handling

Document Format Options:

Simple string array:

{
  "model": "bge-reranker-v2-m3",
  "query": "What is deep learning?",
  "documents": [
    "Deep learning uses neural networks with multiple layers",
    "Machine learning is a subset of artificial intelligence",
    "Natural language processing deals with text understanding"
  ]
}

Structured documents with text field:

{
  "model": "rerank-english-v3.0",
  "query": "What is deep learning?",
  "documents": [
    {"text": "Deep learning uses neural networks with multiple layers"},
    {"text": "Machine learning is a subset of artificial intelligence"}
  ]
}

Response:

{
  "results": [
    {
      "index": 0,
      "relevance_score": 0.95
    },
    {
      "index": 2,
      "relevance_score": 0.72
    },
    {
      "index": 1,
      "relevance_score": 0.45
    }
  ],
  "model": "bge-reranker-v2-m3",
  "id": "rerank-abc123",
  "usage": {
    "prompt_tokens": 150,
    "total_tokens": 150
  }
}

Response with Documents (when return_documents: true):

{
  "results": [
    {
      "index": 0,
      "relevance_score": 0.95,
      "document": {
        "text": "Deep learning uses neural networks with multiple layers"
      }
    }
  ],
  "model": "bge-reranker-v2-m3"
}

Response Fields:

Field Type Description
results array List of reranked results ordered by relevance score (highest first)
results[].index integer The index of the document in the original input list
results[].relevance_score number Relevance score (typically 0.0 to 1.0, higher is more relevant)
results[].document object The document text (only present if return_documents was true)
model string The model used for reranking
id string Unique identifier for this request (optional)
usage object Token usage information (optional)

Supported Backends:

Backend Endpoint Notes
vLLM /v1/rerank Cohere-compatible, supports BGE, Jina rerankers
llama.cpp /v1/rerank Requires --reranking flag at startup
Hugging Face TEI /rerank Text Embeddings Inference server
Cohere API /v1/rerank Native Cohere rerank endpoint
Jina AI /v1/rerank Native Jina rerank endpoint

Status Codes:

  • 200: Documents reranked successfully
  • 400: Invalid request (missing model/query/documents, empty documents array)
  • 401: Invalid API key
  • 404: Model not found or doesn't support reranking
  • 429: Rate limit exceeded
  • 500: Backend error
  • 503: Backend unavailable

Use Cases:

  • Two-stage retrieval: Use vector search to retrieve candidates, then rerank for higher precision
  • RAG systems: Improve context quality by reranking retrieved documents before LLM processing
  • Search result refinement: Reorder search results based on semantic relevance

Example Request:

curl -X POST http://localhost:8080/v1/rerank \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "model": "bge-reranker-v2-m3",
    "query": "What are the benefits of renewable energy?",
    "documents": [
      "Solar panels convert sunlight into electricity",
      "Wind turbines generate power from wind",
      "Coal is a fossil fuel used for electricity"
    ],
    "top_n": 2
  }'

Sparse Embeddings

Generate sparse embeddings for input text using the TEI/Jina-compatible Sparse Embedding API. Sparse embeddings (e.g., SPLADE) preserve lexical information through explicit term weights, complementing dense embeddings for hybrid search.

POST /embed_sparse

Request Body:

{
  "model": "naver/splade-v3",
  "input": "What is sparse embedding?"
}

Request Parameters:

Field Type Required Description
model string Yes ID of the sparse embedding model to use (e.g., naver/splade-v3, naver/splade-cocondenser-ensembledistil)
input string or array Yes Input text to embed. Can be a single string or an array of strings

Input Format Examples:

Single string:

{
  "model": "naver/splade-v3",
  "input": "What is sparse embedding?"
}

Array of strings:

{
  "model": "naver/splade-v3",
  "input": [
    "First text to embed",
    "Second text to embed",
    "Third text to embed"
  ]
}

Response:

{
  "data": [
    {
      "index": 0,
      "sparse_embedding": {
        "indices": [123, 456, 789, 1024, 2048],
        "values": [0.5, 0.3, 0.1, 0.8, 0.2]
      }
    }
  ],
  "model": "naver/splade-v3",
  "usage": {
    "prompt_tokens": 8,
    "total_tokens": 8
  }
}

Response Fields:

Field Type Description
data array Array of sparse embedding objects
data[].index integer Index of the embedding in the input array
data[].sparse_embedding object The sparse embedding vector
data[].sparse_embedding.indices array Indices of non-zero elements in the vocabulary
data[].sparse_embedding.values array Values at the corresponding indices
model string Model used to generate the embedding (optional)
usage object Token usage information (optional)

Understanding Sparse Vectors:

A sparse vector only stores non-zero values along with their vocabulary indices. For example:

  • indices: [123, 456, 789] - positions in the vocabulary
  • values: [0.5, 0.3, 0.1] - weights for those terms

This is memory-efficient for high-dimensional vectors with few non-zero elements (typically 100-500 non-zero values out of 30,000+ vocabulary size).

Supported Backends:

Backend Endpoint Notes
vLLM /embed_sparse Supports SPLADE models via OpenAI-compatible server
Hugging Face TEI /embed_sparse Requires --pooling splade flag
Jina AI Native Native sparse embedding support

Status Codes:

  • 200: Sparse embeddings generated successfully
  • 400: Invalid request (missing model/input, empty input)
  • 401: Invalid API key
  • 404: Model not found or doesn't support sparse embeddings
  • 429: Rate limit exceeded
  • 500: Backend error
  • 503: Backend unavailable

Use Cases:

  • Hybrid search: Combine dense (semantic) and sparse (lexical) retrieval for better results
  • Keyword matching: Exact term matching with learned weights
  • Domain-specific retrieval: Better handling of specialized terminology and rare words
  • Cross-lingual retrieval: Some SPLADE models support multilingual sparse retrieval

Example Request:

curl -X POST http://localhost:8080/embed_sparse \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "model": "naver/splade-v3",
    "input": "What are the benefits of sparse embeddings for search?"
  }'

Example with Multiple Inputs:

curl -X POST http://localhost:8080/embed_sparse \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "model": "naver/splade-v3",
    "input": [
      "First query text",
      "Second query text"
    ]
  }'

Files API

The Files API allows you to upload, manage, and use files in chat completions. Uploaded files can be referenced in messages using the image_file content type, and the router automatically resolves these references by injecting the file content.

Upload File

Upload a file for use in chat completions.

POST /v1/files
Content-Type: multipart/form-data

Form Fields:

Field Type Required Description
file file Yes The file to upload
purpose string Yes Purpose of the file: vision, assistants, fine-tune, batch, user_data, evals

Example:

curl -X POST http://localhost:8080/v1/files \
  -F "file=@image.png" \
  -F "purpose=vision"

Response:

{
  "id": "file-abc123def456",
  "object": "file",
  "bytes": 12345,
  "created_at": 1699061776,
  "filename": "image.png",
  "purpose": "vision"
}

Status Codes:

  • 200: File uploaded successfully
  • 400: Invalid request (missing file, invalid purpose)
  • 413: File too large (exceeds configured max_file_size)

List Files

Retrieve a list of uploaded files.

GET /v1/files
GET /v1/files?purpose=vision

Query Parameters:

Parameter Type Required Description
purpose string No Filter by purpose

Response:

{
  "object": "list",
  "data": [
    {
      "id": "file-abc123def456",
      "object": "file",
      "bytes": 12345,
      "created_at": 1699061776,
      "filename": "image.png",
      "purpose": "vision"
    }
  ]
}


Get File Metadata

Retrieve metadata for a specific file.

GET /v1/files/{file_id}

Response:

{
  "id": "file-abc123def456",
  "object": "file",
  "bytes": 12345,
  "created_at": 1699061776,
  "filename": "image.png",
  "purpose": "vision"
}

Status Codes:

  • 200: File metadata retrieved
  • 404: File not found

Download File Content

Download the content of an uploaded file.

GET /v1/files/{file_id}/content

Response: Binary file content with appropriate Content-Type header.

Status Codes:

  • 200: File content returned
  • 404: File not found

Delete File

Delete an uploaded file.

DELETE /v1/files/{file_id}

Response:

{
  "id": "file-abc123def456",
  "object": "file",
  "deleted": true
}

Status Codes:

  • 200: File deleted successfully
  • 404: File not found

Error Bodies

Every Files API error uses the OpenAI envelope: an error object with message, type, and code.

Client-caused failures name what the request got wrong, including the file_id the caller supplied:

{
  "error": {
    "message": "file not found: file-abc123def456",
    "type": "invalid_request_error",
    "code": "file_not_found"
  }
}

Server-side failures return a fixed message instead of the router's internal diagnostics. The type and code still distinguish the cause, so a client that branches on them is unaffected:

{
  "error": {
    "message": "storage error: the file operation could not be completed",
    "type": "server_error",
    "code": "storage_error"
  }
}

The full diagnostic (the absolute path under files.storage_path, the temporary upload filename, and the underlying operating-system error) is written to the router log at ERROR level and is never placed in a response body. It is deliberately not exposed through a configuration switch: an operator debugging a storage failure reads the log, and a caller has no use for the router's filesystem layout. The same applies to the /admin/files* endpoints and to the file_id resolution failures surfaced on /v1/chat/completions and /anthropic/v1/messages.

code Status Message
storage_error 500 Fixed: storage error: the file operation could not be completed
io_error 500 Fixed: io error: the file operation could not be completed
service_unavailable 503 Fixed: files api is disabled
everything else 4xx Describes the request, including the file_id

File Resolution in Chat Completions

The router automatically resolves file references in chat completion requests. When a message contains an image_file content block, the router:

  1. Validates the file ID format
  2. Loads the file content from storage
  3. Converts the file to a base64 data URL
  4. Replaces the image_file block with an image_url block

Supported File Types

File resolution in chat completions supports the following file types:

File Type MIME Type Support
PNG image/png All backends
JPEG image/jpeg All backends
GIF image/gif All backends
WebP image/webp All backends
PDF application/pdf OpenAI, Anthropic
Plain Text text/plain Anthropic

Note: PDF and plain text support is available for Anthropic backends (and PDF for OpenAI). The file transformers automatically convert document files to the appropriate format for each backend (OpenAI uses file blocks for PDF, Anthropic uses document blocks for both PDF and plain text). Non-image and non-document files will return a 400 Bad Request error with a helpful message indicating the supported file types.

Request with File Reference:

{
  "model": "gpt-4-vision-preview",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What's in this image?"},
        {"type": "image_file", "image_file": {"file_id": "file-abc123def456"}}
      ]
    }
  ]
}

Transformed Request (sent to backend):

{
  "model": "gpt-4-vision-preview",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What's in this image?"},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
      ]
    }
  ]
}

File Resolution Errors:

Error Status Description
Invalid file ID format 400 File ID must start with file-
File not found 404 Referenced file does not exist
Too many file references 400 Request contains more than 20 file references
Resolution timeout 504 File resolution took longer than 30 seconds

Supported MIME Types:

  • image/png - All backends
  • image/jpeg - All backends
  • image/gif - All backends
  • image/webp - All backends
  • application/pdf - OpenAI, Anthropic (max 32MB, 100 pages)
  • text/plain - Anthropic (max 32MB)

Anthropic Native API

Continuum Router provides native Anthropic API endpoints that allow clients to use Anthropic's API format directly while still benefiting from the router's load balancing, failover, and multi-backend routing capabilities.

Messages API

Send messages using Anthropic's native API format.

POST /anthropic/v1/messages

Headers:

Header Required Description
x-api-key or Authorization: Bearer Yes* API key for authentication (required in blocking mode); either header form is accepted, and Authorization: Bearer takes precedence when both are present
anthropic-version No API version (e.g., 2023-06-01). Forwarded to native Anthropic backends
anthropic-beta No Beta features (e.g., prompt-caching-2024-07-31). Forwarded to native Anthropic backends

Request Body:

{
  "model": "claude-sonnet-4-20250514",
  "max_tokens": 1024,
  "messages": [
    {
      "role": "user",
      "content": "Hello, Claude!"
    }
  ],
  "stream": false
}

Request Parameters:

Parameter Type Required Description
model string Yes Model identifier
max_tokens integer Yes Maximum tokens to generate
messages array Yes Array of message objects
system string/array No System prompt (string or array of content blocks)
stream boolean No Enable streaming (default: false)
temperature number No Sampling temperature (0-1)
top_p number No Nucleus sampling parameter
top_k integer No Top-k sampling parameter
stop_sequences array No Stop sequences
metadata object No Request metadata
tools array No Tool definitions for function calling
tool_choice object No Tool choice configuration

Response (non-streaming):

{
  "id": "msg_01XFDUDYJgAACzvnptvVoYEL",
  "type": "message",
  "role": "assistant",
  "content": [
    {
      "type": "text",
      "text": "Hello! How can I help you today?"
    }
  ],
  "model": "claude-sonnet-4-20250514",
  "stop_reason": "end_turn",
  "stop_sequence": null,
  "usage": {
    "input_tokens": 12,
    "output_tokens": 15,
    "cache_creation_input_tokens": 0,
    "cache_read_input_tokens": 0
  }
}

Backend Routing:

The router automatically transforms requests based on the target backend:

Backend Type Transformation
Anthropic Pass-through with native API format
Gemini Direct transformation to Gemini format
OpenAI/vLLM/Ollama Transform to OpenAI format, then transform response back
OpenAI / Azure OpenAI with responses_only model Transform to Responses API format, dispatch to /v1/responses, transform response back to Anthropic Messages JSON or Anthropic SSE events

Transparent Responses-API routing

When the requested model is marked responses_only: true in model-metadata.yaml or the built-in OpenAI registry (e.g., gpt-5-pro, gpt-5.2-pro, gpt-5.4-pro, gpt-5.5-pro), the router transparently routes the Anthropic-formatted request to the upstream /v1/responses endpoint instead of /v1/chat/completions:

  • The request is converted from Anthropic Messages format into the Responses API shape before dispatch.
  • The upstream response is converted back into Anthropic Messages JSON (non-streaming) or the Anthropic SSE event sequence (streaming).
  • Tool-call round-trips, web-search emulation, and Unix-socket transports all honor the flag.

The dispatch is transparent: clients keep targeting /anthropic/v1/messages and continue to receive Anthropic-shaped responses. Only OpenAI and Azure OpenAI backends serve these models; pairing a responses_only model with another backend type produces a 400 invalid_request_error (Anthropic-shaped on this surface) before any upstream call. See Responses-API-only Models for the full list, configuration syntax, and how to mark new models.

File references and resolution limits

Image and document blocks may point at a file uploaded through the Files API instead of carrying inline base64:

{"type": "image", "source": {"type": "file_id", "file_id": "file-abc123def456"}}

Every such reference is loaded and base64-encoded into the request before dispatch, including references nested inside a tool_result content array. Each reference gets its own copy of the encoded data, so the cost of a request grows with the number of references rather than with the number of distinct files. The following per-request limits apply:

Limit Value On violation
References per request 100 400 invalid_request_error, evaluated before any file is read
Aggregate referenced bytes 32 MiB 400 invalid_request_error; counted once per reference, so referencing one 10 MiB file four times costs 40 MiB
File loads in flight 8 No error; additional loads wait for a slot
Total resolution time 30 seconds 504

A single referenced file is additionally capped at 10 MiB, unchanged from earlier releases.

The reference cap is intentionally higher than the 20-reference limit on /v1/chat/completions. The Messages API replays the whole conversation transcript on every turn, so a long multimodal session legitimately accumulates references that a single chat-completions call never would. Base64 expands the raw bytes by roughly 4/3, so the 32 MiB budget corresponds to an emitted ceiling of about 43 MiB.

Token Counting

Count the number of tokens in a message request payload.

POST /anthropic/v1/messages/count_tokens

Request Body:

{
  "model": "claude-sonnet-4-20250514",
  "messages": [
    {
      "role": "user",
      "content": "Hello, how are you?"
    }
  ],
  "system": "You are a helpful assistant."
}

Response:

{
  "input_tokens": 18
}

Tiered Token Counting

Token counting uses different strategies based on the backend type:

Backend Type Strategy Accuracy
Anthropic Native /v1/messages/count_tokens API proxy Exact
llama.cpp Backend /tokenize endpoint proxy Exact
vLLM Backend /tokenize endpoint proxy Exact
Others Character-based estimation (~4 chars/token) Approximate

Note: For backends that don't support native tokenization, the router uses a character-based estimation of approximately 4 characters per token. This provides a reasonable approximation for planning purposes but may not match the exact token count used by the model.

Models List

List available models in Anthropic API format.

GET /anthropic/v1/models

Response:

{
  "data": [
    {
      "id": "claude-sonnet-4-20250514",
      "type": "model",
      "display_name": "Claude Sonnet 4",
      "created_at": "2025-05-14T00:00:00Z"
    },
    {
      "id": "claude-opus-4-20250514",
      "type": "model",
      "display_name": "Claude Opus 4",
      "created_at": "2025-05-14T00:00:00Z"
    }
  ],
  "has_more": false,
  "first_id": "claude-sonnet-4-20250514",
  "last_id": "claude-opus-4-20250514"
}

Claude Code Compatibility

The Anthropic Native API includes full compatibility with Claude Code and other Anthropic API clients that require advanced features.

Prompt Caching

Prompt caching is fully supported through the cache_control field on content blocks:

{
  "model": "claude-sonnet-4-20250514",
  "max_tokens": 1024,
  "system": [
    {
      "type": "text",
      "text": "You are a helpful coding assistant with extensive knowledge...",
      "cache_control": {"type": "ephemeral"}
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Previous context...",
          "cache_control": {"type": "ephemeral"}
        },
        {
          "type": "text",
          "text": "Current question"
        }
      ]
    }
  ]
}

Supported cache_control locations:

  • System prompt text blocks
  • User message text blocks
  • User message image blocks
  • Tool definitions
  • Tool use blocks
  • Tool result blocks

Beta Features Header

The anthropic-beta header is forwarded to native Anthropic backends, enabling beta features:

curl -X POST http://localhost:8080/anthropic/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "anthropic-beta: prompt-caching-2024-07-31,interleaved-thinking-2025-05-14" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-20250514",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Cache Usage in Streaming

When streaming with native Anthropic backends, cache usage information is included in the message_start event:

{
  "type": "message_start",
  "message": {
    "id": "msg_01XFDUDYJgAACzvnptvVoYEL",
    "type": "message",
    "role": "assistant",
    "content": [],
    "model": "claude-sonnet-4-20250514",
    "usage": {
      "input_tokens": 2159,
      "cache_creation_input_tokens": 2048,
      "cache_read_input_tokens": 0
    }
  }
}

Interleaved Thinking

Extended thinking with interleaved output is supported in streaming mode. When the model produces thinking and text content alternately, the streaming events properly represent this interleaved structure with appropriate content_block_start, content_block_delta, and content_block_stop events for each block.


Admin API

Administrative routes are mounted under /admin when the admin feature is compiled. Authentication is disabled unless admin.auth is configured.

Common read operations:

GET /admin/capabilities
GET /admin/health
GET /admin/backends
GET /admin/models
GET /admin/config/full
GET /admin/config/hot-reload-status
GET /admin/stats

Mutation, schema, import/export, API-key, prompt, cache, smart-routing, guardrail, Files, ACP, and feature-gated control-plane routes are documented in the Admin REST API reference. That reference also records the current reload boundaries and the configuration endpoints that must not be used as if they applied a pending change.

Error Handling

Error Response Format

All errors follow a consistent JSON structure:

{
  "error": {
    "message": "Human-readable error description",
    "type": "error_type_identifier",
    "code": 404,
    "details": {
      "additional": "context information"
    }
  }
}

Error Types

Type HTTP Code Description
bad_request 400 Invalid request format or parameters
unauthorized 401 Authentication required (blocking auth mode)
forbidden 403 Access denied (insufficient scope or ownership)
model_not_found 404 Requested model not available
rate_limit_exceeded 429 Rate limit exceeded
internal_error 500 Router internal error
bad_gateway 502 Backend connection/response error
service_unavailable 503 All backends unhealthy
gateway_timeout 504 Backend request timeout

Example Error Responses

Model Not Found:

{
  "error": {
    "message": "Model 'invalid-model' not found on any healthy backend",
    "type": "model_not_found", 
    "code": 404,
    "details": {
      "requested_model": "invalid-model",
      "available_models": ["gpt-4", "gpt-3.5-turbo", "llama2"]
    }
  }
}

Backend Error:

{
  "error": {
    "message": "Failed to connect to backend 'local-ollama'",
    "type": "bad_gateway",
    "code": 502,
    "details": {
      "backend": "local-ollama", 
      "backend_error": "Connection refused"
    }
  }
}

Service Unavailable:

{
  "error": {
    "message": "All backends are currently unhealthy",
    "type": "service_unavailable",
    "code": 503,
    "details": {
      "healthy_backends": 0,
      "total_backends": 3
    }
  }
}

Rate Limiting

When rate_limiting is enabled in the configuration, the router enforces token-bucket limits across several dimensions:

  • Per-client (IP address) limits
  • Per-API-key limits, including per-key overrides
  • Per-backend and global limits
  • Model-specific limits

Requests over a limit receive 429 Too Many Requests with a Retry-After header; responses within limits carry X-RateLimit-* headers. See Rate Limiting for configuration, client identification, bypass options, and storage backends.

Streaming

Server-Sent Events (SSE)

When stream: true is specified, responses are sent as Server-Sent Events with:

Content-Type: text/event-stream Cache-Control: no-cache Connection: keep-alive

SSE Format

data: {"id":"chatcmpl-123","object":"chat.completion.chunk",...}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk",...}

data: [DONE]

SSE Compatibility

The router supports multiple SSE formats for maximum compatibility:

  • Standard Format: data: {...}
  • Spaced Format: data: {...}
  • Mixed Line Endings: Handles \r\n, \n, and \r
  • Empty Lines: Properly processes chunk separators

Connection Management

  • Keep-Alive: Connections are kept open during streaming
  • Timeouts: 5-minute timeout for long-running requests
  • Error Handling: Partial responses include error information
  • Client Disconnection: Gracefully handles client disconnects

Examples

Basic Chat Completion

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ]
  }'

Streaming Chat Completion

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {"role": "user", "content": "Write a short story"}
    ],
    "stream": true,
    "max_tokens": 200
  }'

Text Completion with Parameters

curl -X POST http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo-instruct",
    "prompt": "The future of AI is",
    "max_tokens": 50,
    "temperature": 0.8,
    "top_p": 0.9
  }'

Check Backend Status

curl http://localhost:8080/admin/backends | jq

Monitor Service Health

curl http://localhost:8080/admin/health | jq '.services'

List Available Models

curl http://localhost:8080/v1/models | jq '.data[].id'

Python Client Example

import requests
import json

# Configure the client
BASE_URL = "http://localhost:8080"

def chat_completion(messages, model="gpt-3.5-turbo", stream=False):
    """Send a chat completion request"""
    response = requests.post(
        f"{BASE_URL}/v1/chat/completions",
        headers={"Content-Type": "application/json"},
        json={
            "model": model,
            "messages": messages,
            "stream": stream,
            "temperature": 0.7
        },
        stream=stream
    )

    if stream:
        # Handle streaming response
        for line in response.iter_lines():
            if line:
                line = line.decode('utf-8')
                if line.startswith('data: '):
                    data = line[6:]  # Remove 'data: ' prefix
                    if data == '[DONE]':
                        break
                    try:
                        chunk = json.loads(data)
                        content = chunk['choices'][0]['delta'].get('content', '')
                        if content:
                            print(content, end='', flush=True)
                    except json.JSONDecodeError:
                        continue
        print()  # New line after streaming
    else:
        # Handle non-streaming response
        result = response.json()
        return result['choices'][0]['message']['content']

# Example usage
messages = [
    {"role": "user", "content": "Explain machine learning in simple terms"}
]

print("Streaming response:")
chat_completion(messages, stream=True)

print("\nNon-streaming response:")
response = chat_completion(messages, stream=False)
print(response)

JavaScript/Node.js Client Example

const fetch = require('node-fetch');

const BASE_URL = 'http://localhost:8080';

async function chatCompletion(messages, options = {}) {
    const response = await fetch(`${BASE_URL}/v1/chat/completions`, {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
        },
        body: JSON.stringify({
            model: options.model || 'gpt-3.5-turbo',
            messages: messages,
            stream: options.stream || false,
            temperature: options.temperature || 0.7,
            ...options
        })
    });

    if (options.stream) {
        // Handle streaming response
        const reader = response.body.getReader();
        const decoder = new TextDecoder();

        while (true) {
            const { done, value } = await reader.read();
            if (done) break;

            const chunk = decoder.decode(value);
            const lines = chunk.split('\n');

            for (const line of lines) {
                if (line.startsWith('data: ')) {
                    const data = line.slice(6);
                    if (data === '[DONE]') return;

                    try {
                        const parsed = JSON.parse(data);
                        const content = parsed.choices[0]?.delta?.content;
                        if (content) {
                            process.stdout.write(content);
                        }
                    } catch (e) {
                        // Ignore JSON parse errors
                    }
                }
            }
        }
        console.log(); // New line
    } else {
        const result = await response.json();
        return result.choices[0].message.content;
    }
}

// Example usage
const messages = [
    { role: 'user', content: 'What is the meaning of life?' }
];

// Streaming
console.log('Streaming response:');
await chatCompletion(messages, { stream: true });

// Non-streaming  
console.log('\nNon-streaming response:');
const response = await chatCompletion(messages);
console.log(response);

For deployment configuration and admin endpoint usage, see the Admin REST API Reference. For runtime monitoring, see the Metrics and Error Handling guides.