API Reference¶
Continuum Router exposes an OpenAI-compatible API with additional administrative endpoints for monitoring and management. This document covers all available endpoints, request/response formats, and error handling.
Table of Contents¶
- Overview
- Authentication
- Core API Endpoints
- Health Check
- Version
- List Models
- Proxy a Backend Model List
- Force-Refresh Models
- Get Single Model
- Chat Completions
- Responses API
- Retrieve or Delete a Stored Response
- Responses Compact
- Batch API
- Image Generation
- Image Edit (Inpainting)
- Image Variations
- Text Completions
- Embeddings
- Rerank
- Sparse Embeddings
- Files API
- File Resolution in Chat Completions
- Anthropic Native API
- Admin Endpoints
- Configuration Management API
- Error Handling
- Rate Limiting
- Streaming
- Examples
Overview¶
Base URL¶
Content Type¶
All requests and responses use application/json unless otherwise specified.
OpenAI Compatibility¶
Continuum Router provides OpenAI-compatible API surfaces for:
- Chat completions with streaming
- Text completions
- Embeddings
- Reranking (Cohere-compatible /v1/rerank)
- Sparse embeddings (TEI/Jina-compatible /embed_sparse)
- Image generation
- Image editing/inpainting
- Image variations
- Files API (upload, list, retrieve, delete)
- File resolution in chat completions (image_file references)
- Model listing
- Error response formats
Authentication¶
Continuum Router supports API key authentication with configurable enforcement modes.
Authentication Modes¶
The router supports two authentication modes for API endpoints:
| Mode | Behavior |
|---|---|
permissive (default) |
Requests without API key are allowed. Requests with valid API keys are authenticated and can access user-specific features. |
blocking |
Only authenticated requests are processed. Requests without valid API key receive 401 Unauthorized. |
Configuration¶
api_keys:
# Authentication mode: "permissive" (default) or "blocking"
mode: blocking
# API key definitions
api_keys:
- key: "${API_KEY_1}"
id: "key-production-1"
user_id: "user-admin"
organization_id: "org-main"
scopes: [read, write, files, admin]
Protected Endpoints (when mode is blocking)¶
API-key middleware protects all routes in the core API router: /v1/models*, /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/responses*, /v1/images/*, /v1/rerank, /embed_sparse, /anthropic/v1/messages*, /anthropic/v1/models, and feature-gated /v1/batches* routes.
Note: Health and version endpoints (/health, /version) are always accessible without authentication. Admin, Files, and Metrics endpoints have separate authentication mechanisms.
Making Authenticated Requests¶
Every protected route accepts the API key in either of two header forms: Authorization: Bearer <api_key> or the Anthropic-native x-api-key: <api_key>. Both forms work on every route, not only the /anthropic/* endpoints, so Anthropic SDKs and Claude Code (which set only ANTHROPIC_API_KEY and send it via x-api-key) authenticate the same way OpenAI-style clients do. When a request carries both headers, Authorization: Bearer takes precedence.
POST /v1/chat/completions HTTP/1.1
Authorization: Bearer sk-your-api-key
Content-Type: application/json
{
"model": "gpt-4",
"messages": [{"role": "user", "content": "Hello"}]
}
POST /v1/chat/completions HTTP/1.1
x-api-key: sk-your-api-key
Content-Type: application/json
{
"model": "gpt-4",
"messages": [{"role": "user", "content": "Hello"}]
}
Authentication Errors¶
When authentication fails, the API returns:
{
"error": {
"message": "Missing or invalid API key. Expected Authorization: Bearer <api_key> or x-api-key",
"type": "authentication_error",
"code": "invalid_api_key"
}
}
Status Codes:
401 Unauthorized: Missing or invalid API key
Core API Endpoints¶
Health Check¶
Check the health status of the router service.
Response:
Status Codes:
200: Service is healthy
Version¶
Return the running router version.
Response:
Status Codes:
200: Always returned; no authentication required
List Models¶
Retrieve all available models from all healthy backends.
Response:
{
"object": "list",
"data": [
{
"id": "gpt-4",
"object": "model",
"created": 1677610602,
"owned_by": "openai-compatible",
"permission": [],
"root": "gpt-4",
"parent": null
},
{
"id": "llama2:7b",
"object": "model",
"created": 1677610602,
"owned_by": "local-ollama",
"permission": [],
"root": "llama2:7b",
"parent": null
}
]
}
Status Codes:
200: Models retrieved successfully503: All backends are unhealthy
Features:
- Model Aggregation: Combines models from all healthy backends
- Deduplication: Removes duplicate models across backends
- Caching: An internal 60-second hard TTL with stale-while-revalidate and request coalescing
- Health Awareness: Only includes models from healthy backends
Proxy a Backend Model List¶
Return the raw /v1/models response from one selected eligible backend instead of aggregating all backends. This compatibility endpoint uses normal backend selection, retry, health, and caller backend-visibility rules.
Use GET /v1/models for the router's deduplicated aggregate catalog. Use this endpoint only when a client needs one backend's provider-native model-list payload.
Status Codes:
200: Selected backend response returned502: Backend connection or gateway failure503: No eligible healthy backend
Force-Refresh Models¶
Immediately invalidate the model-aggregation cache and synchronously re-aggregate from all configured backends. Returns the same OpenAI-compatible model list shape as GET /v1/models so the caller can use the response immediately.
This endpoint is designed for interactive "Refresh" actions in desktop clients (e.g. the "Refresh models" button in backend.ai-go). It bypasses the stale-while-revalidate and soft-TTL windows, triggering an upstream fetch from every backend before responding.
Response:
{
"object": "list",
"data": [
{
"id": "gpt-4o",
"object": "model",
"created": 1677610602,
"owned_by": "openai-compatible"
}
]
}
Status Codes:
200: Cache cleared and fresh model list returned429 Too Many Requests: Force-refresh rate limit exceeded (3 requests per 5-second burst window, 12 per minute)
Rate Limiting:
The endpoint applies a tighter budget than the regular GET /v1/models because each call triggers an upstream fetch from every configured backend. The default limits are 3 requests per 5-second window and 12 per minute. Valid API keys receive separate buckets keyed by API key ID. Anonymous callers and requests with invalid bearer tokens share one global anonymous bucket, so spoofed Authorization, X-Forwarded-For, or X-Real-IP headers cannot bypass the force-refresh budget. Clients that exceed the limit receive 429 and should wait before retrying.
Cache policy:
The model-list cache currently uses internal fixed defaults: a 60-second hard TTL, an 80% soft TTL, stale-while-revalidate, request coalescing, and bounded empty-result backoff. These values and force-refresh availability are not fields in the public configuration schema.
Get Single Model¶
Retrieve information about a specific model, including its availability status and optional rich metadata.
Path Parameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
model |
string | Yes | Model identifier (e.g., "gpt-4", "llama2:7b") |
Response (Basic):
{
"id": "gpt-4",
"object": "model",
"created": 1677610602,
"owned_by": "openai",
"available": true
}
Response (With Extended Metadata):
When model metadata is available in model-metadata.yaml, the response includes additional fields:
{
"id": "gpt-4o",
"object": "model",
"created": 1704067200,
"owned_by": "openai",
"available": true,
"supported_methods": ["chat.completions"],
"features": ["chat", "vision", "audio", "code"],
"max_tokens": 16384,
"metadata": {
"display_name": "GPT-4o",
"developer": "OpenAI",
"summary": "Multimodal optimized model with text, image, and audio capabilities.",
"knowledge_cutoff": "2023-10",
"relative_speed": 4,
"pricing": {
"input_tokens": 2.50,
"output_tokens": 10.0
},
"limits": {
"context_window": 128000,
"max_output": 16384
}
}
}
Response Fields:
| Field | Type | Description |
|---|---|---|
id |
string | Model identifier |
object |
string | Object type (always "model") |
created |
integer | Unix timestamp when model was created |
owned_by |
string | Organization that owns/provides the model |
available |
boolean | Whether the model can currently be used (true if at least one healthy backend provides it) |
supported_methods |
array | (Optional) API methods this model supports (e.g., ["chat.completions"], ["images.generations"]) |
features |
array | (Optional) Model capabilities (e.g., ["chat", "vision", "function_calling"]) |
max_tokens |
integer | (Optional) Maximum output tokens for this model |
metadata |
object | (Optional) Rich metadata object with detailed model information |
Metadata Object Fields:
| Field | Type | Description |
|---|---|---|
display_name |
string | Human-readable display name for the model |
developer |
string | Developer or organization that created the model |
summary |
string | Brief summary describing the model's capabilities |
knowledge_cutoff |
string | Knowledge cutoff date (e.g., "2025-01") |
relative_speed |
integer | Relative speed indicator (1-5, where 1 is slowest and 5 is fastest) |
pricing |
object | Pricing information (input_tokens, output_tokens per 1K tokens) |
limits |
object | Model limits (context_window, max_output in tokens) |
Supported Methods Mapping:
The supported_methods field is derived from model capabilities:
| Capability | API Method |
|---|---|
chat, vision, code, reasoning, audio, video, function_calling, tool |
chat.completions |
embedding |
embeddings |
image_generation |
images.generations |
image_edit |
images.edits |
image_variation |
images.variations |
moderation |
moderations |
Status Codes:
200: Model found and information returned404: Model does not exist in any configured backend
Features:
- OpenAI-Compatible: Response format matches OpenAI API with additional extension fields
- Health-Aware: The
availablefield reflects real-time backend health status - Privacy: Does not expose internal backend information
- Optional Metadata: Extended fields are present only when metadata is available
- Rich Metadata: Includes display name, developer, capabilities, and context-window limits to inform client-side model selection
Model Availability Algorithm:
The available field is determined by the following algorithm:
| Condition | Result |
|---|---|
| Health checker enabled + At least one backend providing the model is healthy | true |
| Health checker enabled + All backends are unhealthy | false |
| Health checker disabled + Model's backend list is non-empty | true |
| Health checker disabled + Model's backend list is empty | false |
The algorithm uses short-circuit evaluation: it returns true as soon as the first healthy backend is found, avoiding unnecessary health checks for remaining backends. This optimizes performance when backends are generally healthy.
Performance Optimization:
- Fast Path: If the model cache is valid, lookup is O(n) where n = number of models
- Slow Path: If cache is empty/expired, triggers one-time aggregation with singleflight protection to prevent cache stampede
- Stale-While-Revalidate: Cache serves stale data while refreshing in background
Example Request:
Example Response (Model Available with Full Metadata):
{
"id": "gpt-4o",
"object": "model",
"created": 1704067200,
"owned_by": "openai",
"available": true,
"supported_methods": ["chat.completions"],
"features": ["chat", "vision", "audio", "code"],
"max_tokens": 16384,
"metadata": {
"display_name": "GPT-4o",
"developer": "OpenAI",
"summary": "Multimodal optimized model with text, image, and audio capabilities; faster and cheaper than GPT-4 Turbo.",
"knowledge_cutoff": "2023-10",
"relative_speed": 4,
"pricing": {
"input_tokens": 2.50,
"output_tokens": 10.0
},
"limits": {
"context_window": 128000,
"max_output": 16384
}
}
}
Example Response (Model Without Extended Metadata):
For models without metadata in model-metadata.yaml, only the basic fields are returned:
{
"id": "custom-model",
"object": "model",
"created": 1677610602,
"owned_by": "local",
"available": true
}
Example Response (Model Exists but Unavailable):
{
"id": "gpt-4o",
"object": "model",
"created": 1704067200,
"owned_by": "openai",
"available": false,
"supported_methods": ["chat.completions"],
"features": ["chat", "vision", "audio", "code"],
"max_tokens": 16384,
"metadata": {
"display_name": "GPT-4o",
"developer": "OpenAI"
}
}
Chat Completions¶
Generate chat completions using the OpenAI Chat API format.
Request Body:
{
"model": "gpt-3.5-turbo",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "Explain quantum computing in simple terms."
}
],
"temperature": 0.7,
"max_tokens": 150,
"top_p": 1.0,
"frequency_penalty": 0.0,
"presence_penalty": 0.0,
"stream": false,
"stop": null,
"logit_bias": {},
"user": "user123"
}
Parameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
model |
string | Yes | Model identifier (must be available on at least one healthy backend) |
messages |
array | Yes | Array of message objects with role and content |
temperature |
number | No | Sampling temperature (0.0 to 2.0, default: 1.0) |
max_tokens |
integer | No | Maximum tokens to generate |
top_p |
number | No | Nucleus sampling parameter (0.0 to 1.0) |
frequency_penalty |
number | No | Frequency penalty (-2.0 to 2.0) |
presence_penalty |
number | No | Presence penalty (-2.0 to 2.0) |
stream |
boolean | No | Enable streaming response (default: false) |
stop |
string/array | No | Stop sequences |
logit_bias |
object | No | Token logit bias |
user |
string | No | User identifier for tracking |
reasoning_effort |
string | No | Reasoning effort level for reasoning-capable models. Valid values vary by backend and model (see below). OpenAI values are validated exactly and are never silently downgraded. auto is reserved for adaptive thinking on Anthropic models. |
reasoning |
object | No | Alternative nested format: {"effort": "high"}. Automatically normalized to reasoning_effort. |
Valid reasoning_effort values by backend:
| Backend | Effort Levels | Token Conversion | Notes |
|---|---|---|---|
| OpenAI O-series (o1, o3, o4-mini) | low, medium, high |
Native | Exact model validation |
| OpenAI GPT-5 (including mini/nano) | minimal, low, medium, high |
Native | Original GPT-5 family only |
| OpenAI GPT-5.1 | none, low, medium, high |
Native | minimal is not accepted |
| OpenAI GPT-5.2 through GPT-5.5 | none, low, medium, high, xhigh |
Native | Exact model validation |
| OpenAI GPT-5.6 | none, low, medium, high, xhigh, max |
Native | max is GPT-5.6-only |
| OpenAI GPT-5 Pro | high |
Native via Responses bridge | Responses API only |
| OpenAI GPT-5.⅖.⅘.5 Pro | medium, high, xhigh |
Native via Responses bridge | Responses API only |
| Anthropic Claude (Opus 4.7, Opus 4.6, Sonnet 4.6, etc.) | none, minimal, auto, low, medium, high, xhigh* |
→ budget_tokens or adaptive + output_config.effort |
*xhigh maps to max on Opus 4.6/4.7 and downgrades to high elsewhere; Opus 4.7 drops temperature/top_p/top_k |
| Gemini (3.x, 2.5-pro, 2.x-flash) | none, auto*, minimal, low, medium, high |
Native | none only for Flash; *auto downgraded to medium |
| Generic/llama.cpp | Any | Pass-through | Backend handles validation |
For detailed conversion tables and backend-specific behavior, see Reasoning Effort Architecture.
For OpenAI models, the router applies the same exact model-specific validation to streaming and non-streaming requests. Unsupported values return 400 invalid_request_error; the router never rewrites auto, xhigh, or max to a different effort. Other providers retain their provider-specific conversion behavior shown above.
Response (Non-streaming):
{
"id": "chatcmpl-123456789",
"object": "chat.completion",
"created": 1677652288,
"model": "gpt-3.5-turbo",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Quantum computing uses quantum mechanical phenomena like superposition and entanglement to process information..."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 25,
"completion_tokens": 150,
"total_tokens": 175
}
}
Response (Streaming):
When stream: true, the response uses Server-Sent Events (SSE):
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-3.5-turbo","choices":[{"delta":{"role":"assistant","content":""},"index":0,"finish_reason":null}]}
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-3.5-turbo","choices":[{"delta":{"content":"Quantum"},"index":0,"finish_reason":null}]}
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-3.5-turbo","choices":[{"delta":{"content":" computing"},"index":0,"finish_reason":null}]}
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-3.5-turbo","choices":[{"delta":{},"index":0,"finish_reason":"stop"}]}
data: [DONE]
Status Codes:
200: Completion generated successfully400: Invalid request format or parameters404: Model not found on any healthy backend502: Backend connection error504: Request timeout503: All backends unhealthy
Features:
- Model-Based Routing: Automatically routes to backends serving the requested model
- Load Balancing: Distributes load across healthy backends
- Streaming Support: Real-time response streaming via SSE
- Error Recovery: Automatic retry on transient failures
- Request Deduplication: Prevents duplicate processing of identical requests
- Reasoning Parameter Normalization: Automatically normalizes nested
reasoningformat to flatreasoning_effortformat; removes reasoning parameters for models that don't support them - Transparent Responses-API routing: When the requested model is marked
responses_only: true(e.g.,gpt-5-pro,gpt-5.2-pro,gpt-5.4-pro,gpt-5.5-pro), the router transparently dispatches to the upstream/v1/responsesendpoint and translates the result back into a strict-modechat.completion(orchat.completion.chunkfor streaming) envelope, so clients keep using/v1/chat/completionsunchanged. Only OpenAI and Azure OpenAI backends serve these models; pairing aresponses_onlymodel with another backend type produces a400 invalid_request_errorbefore dispatch. See Responses-API-only Models for the full list and configuration.
Responses API¶
Generate responses using OpenAI's Responses API format. This endpoint provides an alternative interface to Chat Completions, internally converting requests to the Chat Completions format for backend processing.
Request Body:
{
"model": "gpt-4o",
"input": "Explain quantum computing in simple terms.",
"instructions": "You are a helpful assistant that explains complex topics simply.",
"max_output_tokens": 1000,
"temperature": 0.7,
"reasoning": {
"effort": "high"
}
}
Parameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
model |
string | Yes | Model identifier (must be available on at least one healthy backend) |
input |
string/array | Yes | The input text or array of input items |
instructions |
string | No | System instructions for the model (converted to system message) |
max_output_tokens |
integer | No | Maximum tokens to generate |
temperature |
number | No | Sampling temperature (0.0 to 2.0) |
top_p |
number | No | Nucleus sampling parameter (0.0 to 1.0) |
stream |
boolean | No | Enable streaming response (default: false) |
store |
boolean | No | Whether to persist the response in session storage (default: true). Set to false for ephemeral requests that should not be stored (used by Codex) |
include_reasoning |
boolean | No | Include reasoning content in the response |
reasoning |
object | No | Reasoning configuration with nested format (see below) |
tools |
array | No | List of tools available for the model (flat format) |
tool_choice |
string/object | No | Controls tool usage |
previous_response_id |
string | No | Reference to a previous response for multi-turn conversations |
Tool Definition Format:
The Responses API uses a flat tool format where function properties are at the same level as the type field. This differs from the Chat Completions API which uses a nested function object.
{
"type": "function",
"name": "get_weather",
"description": "Get weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "City name"}
},
"required": ["location"]
},
"strict": true
}
| Field | Type | Required | Description |
|---|---|---|---|
type |
string | Yes | Must be "function" |
name |
string | Yes | Name of the function |
description |
string | No | Description of what the function does |
parameters |
object | No | JSON Schema for function parameters |
strict |
boolean | No | Enable strict parameter validation |
Other supported tool types include code_interpreter, file_search, web_search, and computer_use.
Multi-Modal Input Types:
When input is an array of items, each item can be a message containing multi-modal content parts. The following content part types are supported:
| Type | Description |
|---|---|
text |
Plain text content |
input_text |
Text content for user messages (alternative format for Responses API) |
output_text |
Text content for assistant messages; differentiates assistant-generated content from user input (input_text) |
input_file |
File content (PDF, images, or other files) |
input_image |
Image content |
image_url |
Image from URL or base64 data |
Message Roles:
Each message item carries one of five roles. The router accepts all five for /v1/responses and translates them per backend.
| Role | Purpose | Per-backend mapping |
|---|---|---|
developer |
Application-developer instructions for OpenAI reasoning models (GPT-5.x, o-series). Highest priority in the chain of command and replaces system for those models. |
OpenAI: passthrough as developer. Anthropic: merged into the system parameter (concatenated with any system text using \n\n). Gemini: merged into system_instruction. Ollama / vLLM / LocalAI / LM Studio: mapped to system. |
system |
Legacy application instructions. | Behaves as before. |
user |
End-user input. | Behaves as before. |
assistant |
Prior model output. | Behaves as before. |
tool |
Tool-call result returned to the model. | Behaves as before. |
The router preserves arrival order when multiple developer and system messages are present. When the target backend (Anthropic, Gemini) only accepts a single instruction slot, the router concatenates them with a blank-line separator so no instruction text is silently dropped.
Item References:
An input item of the form {"type": "item_reference", "id": "item_..."} points at an output item from a previously stored response. This is how the OpenAI and Vercel AI SDKs submit multi-step tool round-trips by default: instead of echoing the full function-call item back, step 2 references the item produced in step 1.
- On the OpenAI / Azure OpenAI passthrough path, references are forwarded unchanged and the upstream resolves them.
- On the converted paths (Anthropic, Gemini, and chat-completions-backed routes), the router resolves each reference from its session store before dispatch, rewriting it to the inline
function_call,message, orfunction_call_outputitem it points to. References are de-duplicated bycall_id(first wins). - Streamed responses participate as well: the accumulated response is stored when the stream completes, so a follow-up request can reference items produced by a streaming step.
- An unresolvable reference returns HTTP 400 naming the id, and a request may carry at most 256 references.
Input File Format:
The input_file content part supports three input methods:
{
"type": "input_file",
"filename": "document.pdf",
"file_data": "data:application/pdf;base64,JVBERi0xLjQ..."
}
| Field | Type | Description |
|---|---|---|
filename |
string | Optional filename for the file |
file_data |
string | Base64 data URL (e.g., data:application/pdf;base64,...) |
file_url |
string | External URL to the file (validated for SSRF) |
file_id |
string | Reference to a file uploaded via Files API |
Input Image Format:
| Field | Type | Description |
|---|---|---|
image_url |
string | Image URL or base64 data URL |
file_id |
string | Reference to an image file uploaded via Files API |
detail |
string | Image detail level: low, high, or auto (default) |
Exactly one of image_url or file_id must be present. A part with neither is rejected with 400. When file_id is provided, the file is resolved to an inline base64 data URL before the request reaches the backend; file ownership is verified and a 10MB size limit applies, matching the input_file + file_id path.
Multi-Modal Request Example:
curl -X POST http://localhost:8080/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o",
"input": [
{
"type": "message",
"role": "user",
"content": [
{"type": "input_text", "text": "What does this document say?"},
{"type": "input_file", "filename": "report.pdf", "file_data": "data:application/pdf;base64,..."}
]
}
]
}'
Security Notes:
- External URLs in
file_urlare validated to prevent SSRF attacks - Private IP addresses and localhost URLs are rejected
- Only HTTPS URLs are recommended for external files
- The
file_idfield references files uploaded via the Files API. Files are resolved and converted to base64 before sending to backends. File ownership is verified and a 10MB size limit applies for file injection
Reasoning Parameter:
The reasoning parameter controls the reasoning effort level for reasoning-capable models. It uses a nested format:
Valid effort values:
| Value | Description |
|---|---|
none |
Disable reasoning on models that support it |
minimal |
Minimal reasoning on the original GPT-5 family and supported non-OpenAI models |
auto |
Adaptive thinking (Anthropic) |
low |
Minimal reasoning effort, faster responses |
medium |
Balanced reasoning effort |
high |
Maximum standard reasoning effort |
xhigh |
Extended reasoning on supported model families |
max |
Maximum reasoning on GPT-5.6 |
The router automatically converts this nested format to the flat reasoning_effort format used by Chat Completions backends. Invalid effort levels are rejected with a 400 Bad Request error.
Example Request with Reasoning:
curl -X POST http://localhost:8080/v1/responses \
-H "Content-Type: application/json" \
-d '{
"model": "o1",
"input": "Solve this complex mathematical problem step by step.",
"reasoning": {
"effort": "high"
}
}'
Response:
{
"id": "resp_abc123",
"object": "response",
"created_at": 1699000000,
"model": "o1",
"output": [
{
"type": "message",
"id": "msg_001",
"role": "assistant",
"content": [
{
"type": "output_text",
"text": "Let me solve this step by step..."
}
]
}
],
"usage": {
"input_tokens": 25,
"output_tokens": 150
},
"status": "completed"
}
Status Codes:
200: Response generated successfully400: Invalid request format, parameters, or invalid reasoning effort value404: Model not found on any healthy backend502: Backend connection error504: Request timeout503: All backends unhealthy
Features:
- Smart Routing: Routes requests based on backend capabilities - native pass-through for OpenAI/Azure, automatic conversion for others
- Native Pass-through: For OpenAI and Azure OpenAI backends, requests are forwarded directly to
/v1/responsesendpoint, preserving all native features - Automatic Conversion: For other backends (Anthropic, Gemini, vLLM, Ollama, etc.), converts Responses API format to their native format
- Reasoning Support: Full support for reasoning parameter with type-safe validation
- Multi-Backend Support: Works with OpenAI, Anthropic, Gemini, Ollama, vLLM, and other backends
- Streaming Support: Real-time response streaming via SSE when
stream: true - Session Management: Supports multi-turn conversations via
previous_response_id
Routing Strategy:
The router automatically determines the best strategy for each backend:
| Backend Type | Strategy | Description |
|---|---|---|
| OpenAI | Pass-through | Native Responses API support - requests forwarded directly |
| Azure OpenAI | Pass-through | Native Responses API support - requests forwarded directly |
| Anthropic | Native Convert | Converted to native Anthropic Messages API format with full PDF/image support |
| Gemini | Convert | Converted to Gemini generateContent API format |
| vLLM | Convert | Converted to Chat Completions format |
| Ollama | Convert | Converted to Chat Completions format |
| LlamaCpp | Convert | Converted to Chat Completions format |
| Generic | Convert | Converted to Chat Completions format |
Pass-through Benefits:
When using OpenAI or Azure OpenAI backends with pass-through mode:
- Native PDF file support (Chat Completions only supports images)
- Preserved reasoning state between turns for better performance
- Access to built-in tools (web_search, file_search, etc.)
- Better cache utilization (40-80% improvement per OpenAI documentation)
- Full compatibility with the latest OpenAI Responses API features
Anthropic Native Conversion Benefits:
When using Anthropic (Claude) backends with native conversion:
- Native PDF file support via Anthropic's document understanding
- Image file support with automatic format detection
- Extended thinking support for Claude 3+ models
- SSRF protection for external file URLs
- Media type whitelisting for security
Retrieve or Delete a Stored Response¶
Responses created with store: true can be retrieved or deleted by ID. Stored sessions are process-local and are scoped to the authenticated API-key principal (api_keys[].user_id) that created the response. Both Authorization: Bearer <key> and X-API-Key: <key> credentials use the same configured identity namespace. Missing, expired, unauthenticated, or differently scoped requests all return 404.
Migration from X-User-ID Scoping¶
X-User-ID is not an authorization boundary for the Responses API. The router ignores it for stored response access, previous_response_id context reconstruction, item-reference resolution, and local Files API ownership checks, and it is not forwarded to backends. Deployments that previously multiplexed multiple end users behind one shared router API key by setting X-User-ID must migrate to one router API key per isolated user or tenant, or place their own trusted service in front of the router and keep per-user state outside the router's stored Responses session and Files API ownership controls.
A successful GET returns the stored response object. A successful DELETE returns:
Responses Compact¶
Compact a conversation context by summarizing or compressing a sequence of input items. For OpenAI and Azure OpenAI backends, the request is forwarded directly to the backend's native /v1/responses/compact endpoint. Other backends return HTTP 501 (Not Implemented), as context compaction is not natively supported.
Request Body:
{
"model": "gpt-4o",
"instructions": "Summarize the key points from this conversation.",
"input": [
{
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": "What is the capital of France?"}]
},
{
"type": "message",
"role": "assistant",
"content": [{"type": "output_text", "text": "The capital of France is Paris."}]
}
]
}
Parameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
model |
string | Yes | Model identifier |
input |
array | Yes | Array of input items representing the conversation to compact |
instructions |
string | No | Optional instructions to guide the compaction process |
Response:
{
"id": "cmpt_abc123",
"object": "response.compaction",
"created_at": 1699000000,
"output": [
{
"type": "text",
"text": "Compacted conversation summary..."
}
],
"usage": {
"input_tokens": 120,
"output_tokens": 45
}
}
Status Codes:
200: Compaction successful400: Invalid request (empty model, empty input, instructions too long)501: Backend does not support native context compaction502: Backend connection error504: Request timeout
Batch API (control-plane builds)¶
The OpenAI-style Batch API is compiled only with the control-plane Cargo feature. It is absent from default source builds. Official release binaries compile the feature, but the routes remain inert unless control_plane.enabled, policy enforcement, and control_plane.batch.enabled are active and the Hub marks the presented key/tier as batch-eligible.
| Method | Path | Purpose |
|---|---|---|
POST |
/v1/batches |
Submit a provider batch |
GET |
/v1/batches |
List batches owned by the calling key |
GET |
/v1/batches/{batch_id} |
Retrieve owned batch status |
POST |
/v1/batches/{batch_id}/cancel |
Cancel an owned batch |
GET |
/v1/batches/{batch_id}/results |
Return completed results as JSONL |
A hub-synced API key is required. Ownership checks deliberately return the same 404 for unknown and other-key IDs. Results return 409 until available. The selected batch backend may be OpenAI-compatible or native Anthropic; the router presents one common surface. x-continuum-batch: off always opts a request out and cannot be overridden by policy.
Image Generation¶
Generate images using OpenAI's DALL-E, GPT Image models, or Google's Nano Banana (Gemini) models.
Request Body:
{
"model": "dall-e-3",
"prompt": "A serene Japanese garden with cherry blossoms",
"n": 1,
"size": "1024x1024",
"quality": "standard",
"response_format": "url"
}
Parameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
model |
string | Yes | Image model: dall-e-2, dall-e-3, gpt-image-1, gpt-image-1.5, gpt-image-1-mini, nano-banana, or nano-banana-pro |
prompt |
string | Yes | Description of the image to generate |
n |
integer | No | Number of images (1-10, varies by model) |
size |
string | No | Image size (varies by model, see below) |
quality |
string | No | Image quality (varies by model, see below) |
style |
string | No | Image style: vivid or natural (DALL-E 3 only) |
response_format |
string | No | Response format: url or b64_json |
output_format |
string | No | Output file format: png, jpeg, webp (GPT Image models only, default: png) |
output_compression |
integer | No | Compression level 0-100 for jpeg/webp (GPT Image models only) |
background |
string | No | Background: transparent, opaque, auto (GPT Image models only) |
stream |
boolean | No | Enable streaming for partial images (GPT Image models only, default: false) |
partial_images |
integer | No | Number of partial images 0-3 during streaming (GPT Image models only) |
user |
string | No | User identifier for tracking |
Model-specific constraints:
| Model | Sizes | n | Quality | Notes |
|---|---|---|---|---|
dall-e-2 |
256x256, 512x512, 1024x1024 |
1-10 | N/A | Classic DALL-E 2 |
dall-e-3 |
1024x1024, 1792x1024, 1024x1792 |
1 | standard, hd |
High quality with prompt revision |
gpt-image-1 |
1024x1024, 1536x1024, 1024x1536, auto |
1 | low, medium, high, auto |
Latest GPT Image model, supports streaming |
gpt-image-1.5 |
1024x1024, 1536x1024, 1024x1536, auto |
1 | low, medium, high, auto |
4x faster, better text rendering |
gpt-image-1-mini |
1024x1024, 1536x1024, 1024x1536, auto |
1 | low, medium, high, auto |
Cost-effective option |
nano-banana |
256x256 to 1024x1024 |
1-4 | N/A | Gemini 2.5 Flash Image (fast) |
nano-banana-pro |
256x256 to 4096x4096 |
1-4 | N/A | Gemini 2.0 Flash Image (advanced, up to 4K) |
Quality Parameter (GPT Image Models):
For GPT Image models, standard maps to medium and hd maps to high.
| Quality | Description |
|---|---|
low |
Fast generation with lower quality |
medium |
Balanced quality and speed (default) |
high |
Best quality, slower generation |
auto |
Model selects optimal quality |
Output Format Options (GPT Image Models):
| Format | Description | Supports Transparency |
|---|---|---|
png |
Lossless format (default) | Yes |
jpeg |
Lossy format, smaller file size | No |
webp |
Modern format, good compression | Yes |
Note: Transparent background (
background: "transparent") requirespngorwebpformat.
Nano Banana (Gemini) Models:
Nano Banana provides access to Google's Gemini image generation capabilities through an OpenAI-compatible interface:
nano-banana: Maps to Gemini 2.5 Flash Image - fast, general-purpose image generationnano-banana-pro: Maps to Gemini 2.0 Flash Image - advanced model with high-resolution support (up to 4K)
Nano Banana Size Mapping:
The router automatically converts OpenAI-style size parameters to Gemini's aspectRatio and imageSize format:
| OpenAI Size | Gemini aspectRatio | Gemini imageSize | Notes |
|---|---|---|---|
256x256 |
1:1 | 1K | Falls back to Gemini minimum |
512x512 |
1:1 | 1K | Falls back to Gemini minimum |
1024x1024 |
1:1 | 1K | Default |
1536x1024 |
3:2 | 1K | Landscape |
1024x1536 |
2:3 | 1K | Portrait |
1024x1792 |
9:16 | 1K | Tall portrait |
1792x1024 |
16:9 | 1K | Wide landscape |
2048x2048 |
1:1 | 2K | Pro only |
4096x4096 |
1:1 | 4K | Pro only |
auto |
1:1 | 1K | Default fallback |
The conversion sends the following Gemini API structure:
{
"contents": [{"parts": [{"text": "Your prompt"}]}],
"generationConfig": {
"imageConfig": {
"aspectRatio": "3:2",
"imageSize": "1K"
}
}
}
Example Nano Banana Request:
{
"model": "nano-banana",
"prompt": "A white siamese cat with blue eyes, photorealistic",
"n": 1,
"size": "1024x1024",
"response_format": "b64_json"
}
Response:
{
"created": 1677652288,
"data": [
{
"url": "https://oaidalleapiprodscus.blob.core.windows.net/...",
"revised_prompt": "A tranquil Japanese garden featuring..."
}
]
}
Response (with b64_json):
{
"created": 1677652288,
"data": [
{
"b64_json": "/9j/4AAQSkZJRgABAQAA...",
"revised_prompt": "A tranquil Japanese garden featuring..."
}
]
}
Nano Banana Response Notes:
- When using
response_format: "url"with Nano Banana, the image is returned as a data URL (data:image/png;base64,...) since Gemini's native API returns inline base64 data - The
revised_promptfield contains any text response from Gemini describing the generated image
Streaming Image Generation (GPT Image Models):
When stream: true is specified for GPT Image models, the response will be streamed as Server-Sent Events (SSE):
Example Streaming Request:
{
"model": "gpt-image-1",
"prompt": "A beautiful sunset over mountains",
"stream": true,
"partial_images": 2,
"response_format": "b64_json"
}
Streaming Response Format:
data: {"type":"image_generation.partial_image","partial_image_index":0,"b64_json":"...","created":1702345678}
data: {"type":"image_generation.partial_image","partial_image_index":1,"b64_json":"...","created":1702345679}
data: {"type":"image_generation.complete","b64_json":"...","created":1702345680}
data: {"type":"image_generation.usage","usage":{"input_tokens":25,"output_tokens":1024}}
data: {"type":"done"}
SSE Event Types:
| Event Type | Description |
|---|---|
image_generation.partial_image |
Intermediate image during generation |
image_generation.complete |
Final complete image |
image_generation.usage |
Token usage information (for cost tracking) |
done |
Stream completion marker |
Example GPT Image Request with New Options:
{
"model": "gpt-image-1.5",
"prompt": "A white cat with blue eyes, photorealistic",
"size": "auto",
"quality": "high",
"output_format": "webp",
"output_compression": 85,
"background": "transparent",
"response_format": "b64_json"
}
Status Codes:
200: Image(s) generated successfully400: Invalid request (e.g., invalid size for model, n > 1 for DALL-E 3)401: Invalid API key429: Rate limit exceeded500: Backend error503: Gemini backend unavailable (for Nano Banana models)
Timeout Configuration:
Image generation requests use a configurable timeout (default: 3 minutes). See timeouts.request.image_generation in configuration.
Image Edit (Inpainting)¶
Edit existing images using OpenAI's inpainting capabilities. This endpoint allows you to modify specific regions of an image based on a text prompt and optional mask. Supports GPT Image models and DALL-E 2.
Request Parameters (multipart/form-data):
| Parameter | Type | Required | Description |
|---|---|---|---|
image |
file | Yes | The source image to edit (PNG, < 4MB, square) |
prompt |
string | Yes | Description of the desired edit |
mask |
file | No | Mask image indicating edit regions (PNG, same dimensions as image) |
model |
string | No | Model to use (default: gpt-image-1) |
n |
integer | No | Number of images to generate (1-10, default: 1) |
size |
string | No | Output size (model-dependent, default: 1024x1024) |
response_format |
string | No | Response format: url or b64_json (default: url) |
user |
string | No | Unique user identifier for tracking |
Supported Models and Sizes:
| Model | Sizes | Notes |
|---|---|---|
gpt-image-1 |
1024x1024, 1536x1024, 1024x1536, auto |
Latest GPT Image model (recommended) |
gpt-image-1-mini |
1024x1024, 1536x1024, 1024x1536, auto |
Cost-optimized version |
gpt-image-1.5 |
1024x1024, 1536x1024, 1024x1536, auto |
Newest with improved instruction following |
dall-e-2 |
256x256, 512x512, 1024x1024 |
Legacy DALL-E 2 model |
Note: DALL-E 3 and Gemini (nano-banana) do NOT support image editing via this endpoint. Gemini uses semantic masking via natural language, which is incompatible with OpenAI's mask-based editing format.
Image Requirements:
- Format: PNG only
- Size: Less than 4MB
- Dimensions: Must be square (width equals height)
Mask Requirements:
- Format: PNG with alpha channel (RGBA)
- Dimensions: Must match the source image exactly
- Transparent areas: Indicate regions to edit/generate
- Opaque areas: Indicate regions to preserve
Example Request:
curl -X POST http://localhost:8080/v1/images/edits \
-F "image=@source_image.png" \
-F "mask=@mask.png" \
-F "prompt=A sunlit indoor lounge area with a pool containing a flamingo" \
-F "n=1" \
-F "size=1024x1024" \
-F "response_format=url"
Example Request (without mask):
curl -X POST http://localhost:8080/v1/images/edits \
-F "image=@source_image.png" \
-F "prompt=Add a sunset in the background" \
-F "n=1" \
-F "size=512x512"
Response:
{
"created": 1677652288,
"data": [
{
"url": "https://oaidalleapiprodscus.blob.core.windows.net/..."
}
]
}
Response (with b64_json):
Status Codes:
200: Image(s) edited successfully400: Invalid request (e.g., non-square image, invalid size, missing required field)401: Invalid API key503: OpenAI backend unavailable
Error Examples:
Non-square image:
{
"error": {
"message": "Image must be square (800x600 is not square)",
"type": "invalid_request_error",
"param": "image",
"code": "image_not_square"
}
}
Mask dimension mismatch:
{
"error": {
"message": "Mask dimensions (256x256) do not match image dimensions (512x512)",
"type": "invalid_request_error",
"param": "mask",
"code": "dimension_mismatch"
}
}
Unsupported model:
{
"error": {
"message": "Model 'dall-e-3' does not support image editing. Supported models: gpt-image-1, gpt-image-1-mini, gpt-image-1.5, dall-e-2. Note: dall-e-3 does NOT support image editing.",
"type": "invalid_request_error",
"param": "model",
"code": "unsupported_model"
}
}
Notes:
- Supported models:
gpt-image-1,gpt-image-1-mini,gpt-image-1.5,dall-e-2 - DALL-E 3 does NOT support image editing via API
- Gemini (nano-banana) is NOT supported - uses different editing approach (semantic masking)
- When no mask is provided, the entire image may be modified
- The source image should have transparent regions if editing without a mask
- Request timeout uses the image generation timeout configuration
Image Variations¶
Generate variations of an existing image using OpenAI's DALL-E 2 model.
Form Fields:
| Parameter | Type | Required | Description |
|---|---|---|---|
image |
file | Yes | Source image for variations (PNG, < 4MB, must be square) |
model |
string | No | Model to use (default: dall-e-2) |
n |
integer | No | Number of variations to generate (1-10, default: 1) |
size |
string | No | Output size: 256x256, 512x512, 1024x1024 (default: 1024x1024) |
response_format |
string | No | Response format: url or b64_json (default: url) |
user |
string | No | User identifier for tracking |
Example Request:
curl -X POST http://localhost:8080/v1/images/variations \
-F "image=@source_image.png" \
-F "model=dall-e-2" \
-F "n=2" \
-F "size=512x512" \
-F "response_format=url"
Response:
{
"created": 1677652288,
"data": [
{
"url": "https://oaidalleapiprodscus.blob.core.windows.net/..."
},
{
"url": "https://oaidalleapiprodscus.blob.core.windows.net/..."
}
]
}
Response (with b64_json):
Model Support:
| Model | Variations Support | Notes |
|---|---|---|
dall-e-2 |
Yes (native) | Full support, 1-10 variations |
dall-e-3 |
No | Not supported by OpenAI API |
gpt-image-1 |
No | Not supported |
nano-banana |
No | Gemini does not support variations API |
nano-banana-pro |
No | Gemini does not support variations API |
Image Requirements:
- Format: PNG only
- Size: Less than 4MB
- Dimensions: Must be square (width == height)
- Supported input sizes: Any square dimensions (will be processed by the model)
Error Scenarios:
| Error | Status | Description |
|---|---|---|
| Image not PNG | 400 | Only PNG format is supported |
| Image not square | 400 | Image dimensions must be equal |
| Image too large | 400 | Image exceeds 4MB size limit |
| Model not supported | 400 | Requested model doesn't support variations |
| Missing image | 400 | Image field is required |
| Invalid n value | 400 | n must be between 1 and 10 |
| Invalid size | 400 | Size must be one of the supported values |
Status Codes:
200: Variation(s) generated successfully400: Invalid request (invalid format, non-square image, unsupported model)401: Invalid API key429: Rate limit exceeded500: Backend error503: Backend unavailable
Text Completions¶
Generate text completions using the OpenAI Completions API format.
Request Body:
{
"model": "gpt-3.5-turbo-instruct",
"prompt": "Once upon a time in a distant galaxy",
"max_tokens": 100,
"temperature": 0.7,
"top_p": 1.0,
"frequency_penalty": 0.0,
"presence_penalty": 0.0,
"stream": false,
"stop": null,
"logit_bias": {},
"user": "user123"
}
Response:
{
"id": "cmpl-123456789",
"object": "text_completion",
"created": 1677652288,
"model": "gpt-3.5-turbo-instruct",
"choices": [
{
"text": ", there lived a young explorer named Zara who dreamed of discovering new worlds...",
"index": 0,
"finish_reason": "stop",
"logprobs": null
}
],
"usage": {
"prompt_tokens": 10,
"completion_tokens": 90,
"total_tokens": 100
}
}
Status Codes: Same as Chat Completions
Embeddings¶
Generate embeddings for input text using the OpenAI Embeddings API format.
Request Body:
{
"model": "text-embedding-3-small",
"input": "The quick brown fox jumps over the lazy dog",
"encoding_format": "float",
"dimensions": 512,
"user": "user123"
}
Request Parameters:
| Field | Type | Required | Description |
|---|---|---|---|
model |
string | Yes | ID of the model to use (e.g., text-embedding-3-small, gemini-embedding-2) |
input |
string, array, or content parts | Yes | Input to embed. Can be a single string, array of strings, array of token arrays, or an array of multimodal content parts (for Gemini Embedding 2) |
encoding_format |
string | No | Format to return the embeddings: float or base64 (default: float) |
dimensions |
integer | No | Number of dimensions for the output embeddings (supported for text-embedding-3-* and gemini-embedding-2* models) |
user |
string | No | Unique identifier for the end-user |
Input Format Examples:
Single string:
Array of strings:
{
"model": "text-embedding-3-large",
"input": [
"First text to embed",
"Second text to embed",
"Third text to embed"
]
}
Token arrays:
Multimodal content parts (Gemini Embedding 2 only):
{
"model": "gemini-embedding-2",
"input": [
{"type": "text", "text": "A photo of a golden retriever"},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQ..."}}
]
}
Response:
{
"object": "list",
"data": [
{
"object": "embedding",
"index": 0,
"embedding": [0.0023064255, -0.009327292, 0.0045318254, ...]
}
],
"model": "text-embedding-3-small",
"usage": {
"prompt_tokens": 8,
"total_tokens": 8
}
}
Response Fields:
| Field | Type | Description |
|---|---|---|
object |
string | Always "list" |
data |
array | Array of embedding objects |
data[].object |
string | Always "embedding" |
data[].index |
integer | Index of the embedding in the input array |
data[].embedding |
array | Embedding vector (array of floats) |
model |
string | Model used to generate the embedding |
usage |
object | Token usage information |
usage.prompt_tokens |
integer | Number of tokens in the input |
usage.total_tokens |
integer | Total tokens used (same as prompt_tokens for embeddings) |
Supported Models:
| Backend | Model | Notes |
|---|---|---|
| OpenAI | text-embedding-3-small |
1536 dimensions by default, supports dimensions parameter |
| OpenAI | text-embedding-3-large |
3072 dimensions by default, supports dimensions parameter |
| OpenAI | text-embedding-ada-002 |
1536 dimensions, legacy model |
| Gemini | gemini-embedding-2 |
3072 dimensions, multimodal (text, image, audio, video, document) |
| Gemini | gemini-embedding-2-preview |
Preview release of Gemini Embedding 2, multimodal |
| Gemini | text-embedding-004 |
Text-only, via OpenAI-compatible endpoint |
| Self-hosted | bge-m3 |
1024 dimensions, 100+ languages, 8192 context. Supports dense, sparse, and ColBERT retrieval |
| Self-hosted | bge-large-en-v1.5 |
1024 dimensions, English-only, 512 context |
| Self-hosted | multilingual-e5-large |
1024 dimensions, 100+ languages, 514 context |
| vLLM | Deployment-specific | Depends on deployed model |
| llama.cpp | Deployment-specific | Native /v1/embeddings support |
| TEI | Deployment-specific | Hugging Face Text Embeddings Inference server |
| Ollama | Deployment-specific | Via Ollama embedding models |
Status Codes:
200: Embeddings generated successfully400: Invalid request (missing model/input, invalid dimensions)401: Invalid API key404: Model not found or doesn't support embeddings429: Rate limit exceeded500: Backend error503: Backend unavailable
Features:
- Multiple Input Formats: Supports single string, array of strings, token arrays, or multimodal content parts
- Multimodal Embeddings:
gemini-embedding-2andgemini-embedding-2-previewaccept mixed text, image, audio, video, and document parts in a single request - Dimension Control: For
text-embedding-3-*andgemini-embedding-2*models, specify custom dimensions for reduced vector size - Backend Agnostic: Routes to appropriate backend based on model
- Load Balancing: Applies configured load balancing strategy
- Error Handling: Returns
400with a clear message when multimodal input is sent to a text-only model
Multimodal Content Part Types (Gemini Embedding 2 only):
| Type | Fields | Constraints |
|---|---|---|
text |
text (string) |
Must not be empty |
image_url |
image_url.url (data URI or file://file-* reference) |
MIME: image/jpeg, image/png; max 20 MiB; up to 6 images per request |
audio |
audio.data (base64), audio.format |
Formats: mp3, wav; max 20 MiB |
video |
video.data (base64), video.format |
Formats: mp4, mov; max 20 MiB |
document |
document.data (base64), document.format |
Formats: pdf; max 20 MiB |
Example Request:
curl -X POST http://localhost:8080/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-api-key" \
-d '{
"model": "text-embedding-3-small",
"input": "The quick brown fox jumps over the lazy dog",
"encoding_format": "float"
}'
Example with Multiple Inputs:
curl -X POST http://localhost:8080/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-api-key" \
-d '{
"model": "text-embedding-3-large",
"input": [
"First document text",
"Second document text",
"Third document text"
]
}'
Example with Custom Dimensions:
curl -X POST http://localhost:8080/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-api-key" \
-d '{
"model": "text-embedding-3-small",
"input": "Sample text",
"dimensions": 512
}'
Example with Multimodal Input (Gemini Embedding 2):
curl -X POST http://localhost:8080/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-api-key" \
-d '{
"model": "gemini-embedding-2",
"input": [
{"type": "text", "text": "A photo of a sunset over the ocean"},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQ..."}}
],
"dimensions": 768
}'
Rerank¶
Rerank documents based on their relevance to a query using the Cohere-compatible Rerank API. This is commonly used as a second-stage retrieval step after initial vector search to improve accuracy.
Request Body:
{
"model": "bge-reranker-v2-m3",
"query": "What is machine learning?",
"documents": ["Document 1 content", "Document 2 content", "Document 3 content"],
"top_n": 3,
"return_documents": false
}
Request Parameters:
| Field | Type | Required | Description |
|---|---|---|---|
model |
string | Yes | ID of the reranking model to use (e.g., bge-reranker-v2-m3, rerank-english-v3.0, jina-reranker-v2-base-multilingual) |
query |
string | Yes | The search query to compare documents against |
documents |
array | Yes | List of documents to rerank. Can be an array of strings or an array of objects with a text field |
top_n |
integer | No | Number of top results to return. If not specified, returns all documents ranked |
return_documents |
boolean | No | Whether to return the document text in the response (default: false) |
max_chunks_per_doc |
integer | No | Maximum number of chunks to process per document for long document handling |
Document Format Options:
Simple string array:
{
"model": "bge-reranker-v2-m3",
"query": "What is deep learning?",
"documents": [
"Deep learning uses neural networks with multiple layers",
"Machine learning is a subset of artificial intelligence",
"Natural language processing deals with text understanding"
]
}
Structured documents with text field:
{
"model": "rerank-english-v3.0",
"query": "What is deep learning?",
"documents": [
{"text": "Deep learning uses neural networks with multiple layers"},
{"text": "Machine learning is a subset of artificial intelligence"}
]
}
Response:
{
"results": [
{
"index": 0,
"relevance_score": 0.95
},
{
"index": 2,
"relevance_score": 0.72
},
{
"index": 1,
"relevance_score": 0.45
}
],
"model": "bge-reranker-v2-m3",
"id": "rerank-abc123",
"usage": {
"prompt_tokens": 150,
"total_tokens": 150
}
}
Response with Documents (when return_documents: true):
{
"results": [
{
"index": 0,
"relevance_score": 0.95,
"document": {
"text": "Deep learning uses neural networks with multiple layers"
}
}
],
"model": "bge-reranker-v2-m3"
}
Response Fields:
| Field | Type | Description |
|---|---|---|
results |
array | List of reranked results ordered by relevance score (highest first) |
results[].index |
integer | The index of the document in the original input list |
results[].relevance_score |
number | Relevance score (typically 0.0 to 1.0, higher is more relevant) |
results[].document |
object | The document text (only present if return_documents was true) |
model |
string | The model used for reranking |
id |
string | Unique identifier for this request (optional) |
usage |
object | Token usage information (optional) |
Supported Backends:
| Backend | Endpoint | Notes |
|---|---|---|
| vLLM | /v1/rerank |
Cohere-compatible, supports BGE, Jina rerankers |
| llama.cpp | /v1/rerank |
Requires --reranking flag at startup |
| Hugging Face TEI | /rerank |
Text Embeddings Inference server |
| Cohere API | /v1/rerank |
Native Cohere rerank endpoint |
| Jina AI | /v1/rerank |
Native Jina rerank endpoint |
Status Codes:
200: Documents reranked successfully400: Invalid request (missing model/query/documents, empty documents array)401: Invalid API key404: Model not found or doesn't support reranking429: Rate limit exceeded500: Backend error503: Backend unavailable
Use Cases:
- Two-stage retrieval: Use vector search to retrieve candidates, then rerank for higher precision
- RAG systems: Improve context quality by reranking retrieved documents before LLM processing
- Search result refinement: Reorder search results based on semantic relevance
Example Request:
curl -X POST http://localhost:8080/v1/rerank \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-api-key" \
-d '{
"model": "bge-reranker-v2-m3",
"query": "What are the benefits of renewable energy?",
"documents": [
"Solar panels convert sunlight into electricity",
"Wind turbines generate power from wind",
"Coal is a fossil fuel used for electricity"
],
"top_n": 2
}'
Sparse Embeddings¶
Generate sparse embeddings for input text using the TEI/Jina-compatible Sparse Embedding API. Sparse embeddings (e.g., SPLADE) preserve lexical information through explicit term weights, complementing dense embeddings for hybrid search.
Request Body:
Request Parameters:
| Field | Type | Required | Description |
|---|---|---|---|
model |
string | Yes | ID of the sparse embedding model to use (e.g., naver/splade-v3, naver/splade-cocondenser-ensembledistil) |
input |
string or array | Yes | Input text to embed. Can be a single string or an array of strings |
Input Format Examples:
Single string:
Array of strings:
{
"model": "naver/splade-v3",
"input": [
"First text to embed",
"Second text to embed",
"Third text to embed"
]
}
Response:
{
"data": [
{
"index": 0,
"sparse_embedding": {
"indices": [123, 456, 789, 1024, 2048],
"values": [0.5, 0.3, 0.1, 0.8, 0.2]
}
}
],
"model": "naver/splade-v3",
"usage": {
"prompt_tokens": 8,
"total_tokens": 8
}
}
Response Fields:
| Field | Type | Description |
|---|---|---|
data |
array | Array of sparse embedding objects |
data[].index |
integer | Index of the embedding in the input array |
data[].sparse_embedding |
object | The sparse embedding vector |
data[].sparse_embedding.indices |
array | Indices of non-zero elements in the vocabulary |
data[].sparse_embedding.values |
array | Values at the corresponding indices |
model |
string | Model used to generate the embedding (optional) |
usage |
object | Token usage information (optional) |
Understanding Sparse Vectors:
A sparse vector only stores non-zero values along with their vocabulary indices. For example:
indices: [123, 456, 789]- positions in the vocabularyvalues: [0.5, 0.3, 0.1]- weights for those terms
This is memory-efficient for high-dimensional vectors with few non-zero elements (typically 100-500 non-zero values out of 30,000+ vocabulary size).
Supported Backends:
| Backend | Endpoint | Notes |
|---|---|---|
| vLLM | /embed_sparse |
Supports SPLADE models via OpenAI-compatible server |
| Hugging Face TEI | /embed_sparse |
Requires --pooling splade flag |
| Jina AI | Native | Native sparse embedding support |
Status Codes:
200: Sparse embeddings generated successfully400: Invalid request (missing model/input, empty input)401: Invalid API key404: Model not found or doesn't support sparse embeddings429: Rate limit exceeded500: Backend error503: Backend unavailable
Use Cases:
- Hybrid search: Combine dense (semantic) and sparse (lexical) retrieval for better results
- Keyword matching: Exact term matching with learned weights
- Domain-specific retrieval: Better handling of specialized terminology and rare words
- Cross-lingual retrieval: Some SPLADE models support multilingual sparse retrieval
Example Request:
curl -X POST http://localhost:8080/embed_sparse \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-api-key" \
-d '{
"model": "naver/splade-v3",
"input": "What are the benefits of sparse embeddings for search?"
}'
Example with Multiple Inputs:
curl -X POST http://localhost:8080/embed_sparse \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-api-key" \
-d '{
"model": "naver/splade-v3",
"input": [
"First query text",
"Second query text"
]
}'
Files API¶
The Files API allows you to upload, manage, and use files in chat completions. Uploaded files can be referenced in messages using the image_file content type, and the router automatically resolves these references by injecting the file content.
Upload File¶
Upload a file for use in chat completions.
Form Fields:
| Field | Type | Required | Description |
|---|---|---|---|
file |
file | Yes | The file to upload |
purpose |
string | Yes | Purpose of the file: vision, assistants, fine-tune, batch, user_data, evals |
Example:
Response:
{
"id": "file-abc123def456",
"object": "file",
"bytes": 12345,
"created_at": 1699061776,
"filename": "image.png",
"purpose": "vision"
}
Status Codes:
200: File uploaded successfully400: Invalid request (missing file, invalid purpose)413: File too large (exceeds configured max_file_size)
List Files¶
Retrieve a list of uploaded files.
Query Parameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
purpose |
string | No | Filter by purpose |
Response:
{
"object": "list",
"data": [
{
"id": "file-abc123def456",
"object": "file",
"bytes": 12345,
"created_at": 1699061776,
"filename": "image.png",
"purpose": "vision"
}
]
}
Get File Metadata¶
Retrieve metadata for a specific file.
Response:
{
"id": "file-abc123def456",
"object": "file",
"bytes": 12345,
"created_at": 1699061776,
"filename": "image.png",
"purpose": "vision"
}
Status Codes:
200: File metadata retrieved404: File not found
Download File Content¶
Download the content of an uploaded file.
Response: Binary file content with appropriate Content-Type header.
Status Codes:
200: File content returned404: File not found
Delete File¶
Delete an uploaded file.
Response:
Status Codes:
200: File deleted successfully404: File not found
Error Bodies¶
Every Files API error uses the OpenAI envelope: an error object with message, type, and code.
Client-caused failures name what the request got wrong, including the file_id the caller supplied:
{
"error": {
"message": "file not found: file-abc123def456",
"type": "invalid_request_error",
"code": "file_not_found"
}
}
Server-side failures return a fixed message instead of the router's internal diagnostics. The type and code still distinguish the cause, so a client that branches on them is unaffected:
{
"error": {
"message": "storage error: the file operation could not be completed",
"type": "server_error",
"code": "storage_error"
}
}
The full diagnostic (the absolute path under files.storage_path, the temporary upload filename, and the underlying operating-system error) is written to the router log at ERROR level and is never placed in a response body. It is deliberately not exposed through a configuration switch: an operator debugging a storage failure reads the log, and a caller has no use for the router's filesystem layout. The same applies to the /admin/files* endpoints and to the file_id resolution failures surfaced on /v1/chat/completions and /anthropic/v1/messages.
code |
Status | Message |
|---|---|---|
storage_error |
500 | Fixed: storage error: the file operation could not be completed |
io_error |
500 | Fixed: io error: the file operation could not be completed |
service_unavailable |
503 | Fixed: files api is disabled |
| everything else | 4xx | Describes the request, including the file_id |
File Resolution in Chat Completions¶
The router automatically resolves file references in chat completion requests. When a message contains an image_file content block, the router:
- Validates the file ID format
- Loads the file content from storage
- Converts the file to a base64 data URL
- Replaces the
image_fileblock with animage_urlblock
Supported File Types¶
File resolution in chat completions supports the following file types:
| File Type | MIME Type | Support |
|---|---|---|
| PNG | image/png |
All backends |
| JPEG | image/jpeg |
All backends |
| GIF | image/gif |
All backends |
| WebP | image/webp |
All backends |
application/pdf |
OpenAI, Anthropic | |
| Plain Text | text/plain |
Anthropic |
Note: PDF and plain text support is available for Anthropic backends (and PDF for OpenAI). The file transformers automatically convert document files to the appropriate format for each backend (OpenAI uses
fileblocks for PDF, Anthropic usesdocumentblocks for both PDF and plain text). Non-image and non-document files will return a400 Bad Requesterror with a helpful message indicating the supported file types.
Request with File Reference:
{
"model": "gpt-4-vision-preview",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{"type": "image_file", "image_file": {"file_id": "file-abc123def456"}}
]
}
]
}
Transformed Request (sent to backend):
{
"model": "gpt-4-vision-preview",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
]
}
]
}
File Resolution Errors:
| Error | Status | Description |
|---|---|---|
| Invalid file ID format | 400 | File ID must start with file- |
| File not found | 404 | Referenced file does not exist |
| Too many file references | 400 | Request contains more than 20 file references |
| Resolution timeout | 504 | File resolution took longer than 30 seconds |
Supported MIME Types:
image/png- All backendsimage/jpeg- All backendsimage/gif- All backendsimage/webp- All backendsapplication/pdf- OpenAI, Anthropic (max 32MB, 100 pages)text/plain- Anthropic (max 32MB)
Anthropic Native API¶
Continuum Router provides native Anthropic API endpoints that allow clients to use Anthropic's API format directly while still benefiting from the router's load balancing, failover, and multi-backend routing capabilities.
Messages API¶
Send messages using Anthropic's native API format.
Headers:
| Header | Required | Description |
|---|---|---|
x-api-key or Authorization: Bearer |
Yes* | API key for authentication (required in blocking mode); either header form is accepted, and Authorization: Bearer takes precedence when both are present |
anthropic-version |
No | API version (e.g., 2023-06-01). Forwarded to native Anthropic backends |
anthropic-beta |
No | Beta features (e.g., prompt-caching-2024-07-31). Forwarded to native Anthropic backends |
Request Body:
{
"model": "claude-sonnet-4-20250514",
"max_tokens": 1024,
"messages": [
{
"role": "user",
"content": "Hello, Claude!"
}
],
"stream": false
}
Request Parameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
model |
string | Yes | Model identifier |
max_tokens |
integer | Yes | Maximum tokens to generate |
messages |
array | Yes | Array of message objects |
system |
string/array | No | System prompt (string or array of content blocks) |
stream |
boolean | No | Enable streaming (default: false) |
temperature |
number | No | Sampling temperature (0-1) |
top_p |
number | No | Nucleus sampling parameter |
top_k |
integer | No | Top-k sampling parameter |
stop_sequences |
array | No | Stop sequences |
metadata |
object | No | Request metadata |
tools |
array | No | Tool definitions for function calling |
tool_choice |
object | No | Tool choice configuration |
Response (non-streaming):
{
"id": "msg_01XFDUDYJgAACzvnptvVoYEL",
"type": "message",
"role": "assistant",
"content": [
{
"type": "text",
"text": "Hello! How can I help you today?"
}
],
"model": "claude-sonnet-4-20250514",
"stop_reason": "end_turn",
"stop_sequence": null,
"usage": {
"input_tokens": 12,
"output_tokens": 15,
"cache_creation_input_tokens": 0,
"cache_read_input_tokens": 0
}
}
Backend Routing:
The router automatically transforms requests based on the target backend:
| Backend Type | Transformation |
|---|---|
| Anthropic | Pass-through with native API format |
| Gemini | Direct transformation to Gemini format |
| OpenAI/vLLM/Ollama | Transform to OpenAI format, then transform response back |
OpenAI / Azure OpenAI with responses_only model |
Transform to Responses API format, dispatch to /v1/responses, transform response back to Anthropic Messages JSON or Anthropic SSE events |
Transparent Responses-API routing¶
When the requested model is marked responses_only: true in model-metadata.yaml or the built-in OpenAI registry (e.g., gpt-5-pro, gpt-5.2-pro, gpt-5.4-pro, gpt-5.5-pro), the router transparently routes the Anthropic-formatted request to the upstream /v1/responses endpoint instead of /v1/chat/completions:
- The request is converted from Anthropic Messages format into the Responses API shape before dispatch.
- The upstream response is converted back into Anthropic Messages JSON (non-streaming) or the Anthropic SSE event sequence (streaming).
- Tool-call round-trips, web-search emulation, and Unix-socket transports all honor the flag.
The dispatch is transparent: clients keep targeting /anthropic/v1/messages and continue to receive Anthropic-shaped responses. Only OpenAI and Azure OpenAI backends serve these models; pairing a responses_only model with another backend type produces a 400 invalid_request_error (Anthropic-shaped on this surface) before any upstream call. See Responses-API-only Models for the full list, configuration syntax, and how to mark new models.
File references and resolution limits¶
Image and document blocks may point at a file uploaded through the Files API instead of carrying inline base64:
Every such reference is loaded and base64-encoded into the request before dispatch, including references nested inside a tool_result content array. Each reference gets its own copy of the encoded data, so the cost of a request grows with the number of references rather than with the number of distinct files. The following per-request limits apply:
| Limit | Value | On violation |
|---|---|---|
| References per request | 100 | 400 invalid_request_error, evaluated before any file is read |
| Aggregate referenced bytes | 32 MiB | 400 invalid_request_error; counted once per reference, so referencing one 10 MiB file four times costs 40 MiB |
| File loads in flight | 8 | No error; additional loads wait for a slot |
| Total resolution time | 30 seconds | 504 |
A single referenced file is additionally capped at 10 MiB, unchanged from earlier releases.
The reference cap is intentionally higher than the 20-reference limit on /v1/chat/completions. The Messages API replays the whole conversation transcript on every turn, so a long multimodal session legitimately accumulates references that a single chat-completions call never would. Base64 expands the raw bytes by roughly 4/3, so the 32 MiB budget corresponds to an emitted ceiling of about 43 MiB.
Token Counting¶
Count the number of tokens in a message request payload.
Request Body:
{
"model": "claude-sonnet-4-20250514",
"messages": [
{
"role": "user",
"content": "Hello, how are you?"
}
],
"system": "You are a helpful assistant."
}
Response:
Tiered Token Counting¶
Token counting uses different strategies based on the backend type:
| Backend Type | Strategy | Accuracy |
|---|---|---|
| Anthropic | Native /v1/messages/count_tokens API proxy |
Exact |
| llama.cpp | Backend /tokenize endpoint proxy |
Exact |
| vLLM | Backend /tokenize endpoint proxy |
Exact |
| Others | Character-based estimation (~4 chars/token) | Approximate |
Note: For backends that don't support native tokenization, the router uses a character-based estimation of approximately 4 characters per token. This provides a reasonable approximation for planning purposes but may not match the exact token count used by the model.
Models List¶
List available models in Anthropic API format.
Response:
{
"data": [
{
"id": "claude-sonnet-4-20250514",
"type": "model",
"display_name": "Claude Sonnet 4",
"created_at": "2025-05-14T00:00:00Z"
},
{
"id": "claude-opus-4-20250514",
"type": "model",
"display_name": "Claude Opus 4",
"created_at": "2025-05-14T00:00:00Z"
}
],
"has_more": false,
"first_id": "claude-sonnet-4-20250514",
"last_id": "claude-opus-4-20250514"
}
Claude Code Compatibility¶
The Anthropic Native API includes full compatibility with Claude Code and other Anthropic API clients that require advanced features.
Prompt Caching¶
Prompt caching is fully supported through the cache_control field on content blocks:
{
"model": "claude-sonnet-4-20250514",
"max_tokens": 1024,
"system": [
{
"type": "text",
"text": "You are a helpful coding assistant with extensive knowledge...",
"cache_control": {"type": "ephemeral"}
}
],
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Previous context...",
"cache_control": {"type": "ephemeral"}
},
{
"type": "text",
"text": "Current question"
}
]
}
]
}
Supported cache_control locations:
- System prompt text blocks
- User message text blocks
- User message image blocks
- Tool definitions
- Tool use blocks
- Tool result blocks
Beta Features Header¶
The anthropic-beta header is forwarded to native Anthropic backends, enabling beta features:
curl -X POST http://localhost:8080/anthropic/v1/messages \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "anthropic-beta: prompt-caching-2024-07-31,interleaved-thinking-2025-05-14" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-sonnet-4-20250514",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]
}'
Cache Usage in Streaming¶
When streaming with native Anthropic backends, cache usage information is included in the message_start event:
{
"type": "message_start",
"message": {
"id": "msg_01XFDUDYJgAACzvnptvVoYEL",
"type": "message",
"role": "assistant",
"content": [],
"model": "claude-sonnet-4-20250514",
"usage": {
"input_tokens": 2159,
"cache_creation_input_tokens": 2048,
"cache_read_input_tokens": 0
}
}
}
Interleaved Thinking¶
Extended thinking with interleaved output is supported in streaming mode. When the model produces thinking and text content alternately, the streaming events properly represent this interleaved structure with appropriate content_block_start, content_block_delta, and content_block_stop events for each block.
Admin API¶
Administrative routes are mounted under /admin when the admin feature is compiled. Authentication is disabled unless admin.auth is configured.
Common read operations:
GET /admin/capabilities
GET /admin/health
GET /admin/backends
GET /admin/models
GET /admin/config/full
GET /admin/config/hot-reload-status
GET /admin/stats
Mutation, schema, import/export, API-key, prompt, cache, smart-routing, guardrail, Files, ACP, and feature-gated control-plane routes are documented in the Admin REST API reference. That reference also records the current reload boundaries and the configuration endpoints that must not be used as if they applied a pending change.
Error Handling¶
Error Response Format¶
All errors follow a consistent JSON structure:
{
"error": {
"message": "Human-readable error description",
"type": "error_type_identifier",
"code": 404,
"details": {
"additional": "context information"
}
}
}
Error Types¶
| Type | HTTP Code | Description |
|---|---|---|
bad_request |
400 | Invalid request format or parameters |
unauthorized |
401 | Authentication required (blocking auth mode) |
forbidden |
403 | Access denied (insufficient scope or ownership) |
model_not_found |
404 | Requested model not available |
rate_limit_exceeded |
429 | Rate limit exceeded |
internal_error |
500 | Router internal error |
bad_gateway |
502 | Backend connection/response error |
service_unavailable |
503 | All backends unhealthy |
gateway_timeout |
504 | Backend request timeout |
Example Error Responses¶
Model Not Found:
{
"error": {
"message": "Model 'invalid-model' not found on any healthy backend",
"type": "model_not_found",
"code": 404,
"details": {
"requested_model": "invalid-model",
"available_models": ["gpt-4", "gpt-3.5-turbo", "llama2"]
}
}
}
Backend Error:
{
"error": {
"message": "Failed to connect to backend 'local-ollama'",
"type": "bad_gateway",
"code": 502,
"details": {
"backend": "local-ollama",
"backend_error": "Connection refused"
}
}
}
Service Unavailable:
{
"error": {
"message": "All backends are currently unhealthy",
"type": "service_unavailable",
"code": 503,
"details": {
"healthy_backends": 0,
"total_backends": 3
}
}
}
Rate Limiting¶
When rate_limiting is enabled in the configuration, the router enforces token-bucket limits across several dimensions:
- Per-client (IP address) limits
- Per-API-key limits, including per-key overrides
- Per-backend and global limits
- Model-specific limits
Requests over a limit receive 429 Too Many Requests with a Retry-After header; responses within limits carry X-RateLimit-* headers. See Rate Limiting for configuration, client identification, bypass options, and storage backends.
Streaming¶
Server-Sent Events (SSE)¶
When stream: true is specified, responses are sent as Server-Sent Events with:
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
SSE Format¶
data: {"id":"chatcmpl-123","object":"chat.completion.chunk",...}
data: {"id":"chatcmpl-123","object":"chat.completion.chunk",...}
data: [DONE]
SSE Compatibility¶
The router supports multiple SSE formats for maximum compatibility:
- Standard Format:
data: {...} - Spaced Format:
data: {...} - Mixed Line Endings: Handles
\r\n,\n, and\r - Empty Lines: Properly processes chunk separators
Connection Management¶
- Keep-Alive: Connections are kept open during streaming
- Timeouts: 5-minute timeout for long-running requests
- Error Handling: Partial responses include error information
- Client Disconnection: Gracefully handles client disconnects
Examples¶
Basic Chat Completion¶
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "Hello, how are you?"}
]
}'
Streaming Chat Completion¶
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "Write a short story"}
],
"stream": true,
"max_tokens": 200
}'
Text Completion with Parameters¶
curl -X POST http://localhost:8080/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-3.5-turbo-instruct",
"prompt": "The future of AI is",
"max_tokens": 50,
"temperature": 0.8,
"top_p": 0.9
}'
Check Backend Status¶
Monitor Service Health¶
List Available Models¶
Python Client Example¶
import requests
import json
# Configure the client
BASE_URL = "http://localhost:8080"
def chat_completion(messages, model="gpt-3.5-turbo", stream=False):
"""Send a chat completion request"""
response = requests.post(
f"{BASE_URL}/v1/chat/completions",
headers={"Content-Type": "application/json"},
json={
"model": model,
"messages": messages,
"stream": stream,
"temperature": 0.7
},
stream=stream
)
if stream:
# Handle streaming response
for line in response.iter_lines():
if line:
line = line.decode('utf-8')
if line.startswith('data: '):
data = line[6:] # Remove 'data: ' prefix
if data == '[DONE]':
break
try:
chunk = json.loads(data)
content = chunk['choices'][0]['delta'].get('content', '')
if content:
print(content, end='', flush=True)
except json.JSONDecodeError:
continue
print() # New line after streaming
else:
# Handle non-streaming response
result = response.json()
return result['choices'][0]['message']['content']
# Example usage
messages = [
{"role": "user", "content": "Explain machine learning in simple terms"}
]
print("Streaming response:")
chat_completion(messages, stream=True)
print("\nNon-streaming response:")
response = chat_completion(messages, stream=False)
print(response)
JavaScript/Node.js Client Example¶
const fetch = require('node-fetch');
const BASE_URL = 'http://localhost:8080';
async function chatCompletion(messages, options = {}) {
const response = await fetch(`${BASE_URL}/v1/chat/completions`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: options.model || 'gpt-3.5-turbo',
messages: messages,
stream: options.stream || false,
temperature: options.temperature || 0.7,
...options
})
});
if (options.stream) {
// Handle streaming response
const reader = response.body.getReader();
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) break;
const chunk = decoder.decode(value);
const lines = chunk.split('\n');
for (const line of lines) {
if (line.startsWith('data: ')) {
const data = line.slice(6);
if (data === '[DONE]') return;
try {
const parsed = JSON.parse(data);
const content = parsed.choices[0]?.delta?.content;
if (content) {
process.stdout.write(content);
}
} catch (e) {
// Ignore JSON parse errors
}
}
}
}
console.log(); // New line
} else {
const result = await response.json();
return result.choices[0].message.content;
}
}
// Example usage
const messages = [
{ role: 'user', content: 'What is the meaning of life?' }
];
// Streaming
console.log('Streaming response:');
await chatCompletion(messages, { stream: true });
// Non-streaming
console.log('\nNon-streaming response:');
const response = await chatCompletion(messages);
console.log(response);
For deployment configuration and admin endpoint usage, see the Admin REST API Reference. For runtime monitoring, see the Metrics and Error Handling guides.