Skip to content

Router-Managed Web Search

Continuum Router can transparently provide an OpenAI-style web_search tool to chat completion requests routed to self-hosted LLM backends (vLLM, SGLang, Ollama, llama.cpp, MLxcel, LM Studio, Generic OpenAI-compatible, and Continuum Router remote). This lets you run agentic workflows on self-hosted models without wiring search integration into every client.

Commercial providers (OpenAI, Azure OpenAI, Gemini, Anthropic) already ship native web search tools, so the router leaves those requests unchanged. A request to gpt-5.6-sol or claude-sonnet-4.6 flows through the router without tool injection.

How it works

For each chat completion, the router runs the following flow:

  1. Resolves which backend will serve the model.
  2. If the backend is self-hosted and web_search is enabled, injects a web_search function tool into the outbound tools[] array.
  3. Dispatches the request. If the model responds with a tool_calls entry for web_search, the router:
    • parses the {"query": "..."} arguments,
    • calls the configured search provider,
    • appends a tool-role message with the JSON-encoded results, and
    • re-invokes the backend with the enriched conversation.
  4. Repeats up to max_tool_iterations times (default 5), then returns the model's final response.

If the search provider fails mid-execution, the router returns a structured {"error": "..."} tool result to the model instead of failing the HTTP request, allowing the model to apologize, retry, or answer without search.

Anthropic Messages API support

When a client connects through /anthropic/v1/messages, the router supports two web_search tool shapes:

Custom web_search tool

If the client sends a client-defined tool named web_search (an Anthropic custom tool with a JSON input_schema), the router handles it like any other self-hosted request: it translates the request to OpenAI format, runs the bounded tool-execution loop, and translates the response back to Anthropic format. Claude Code uses this path for its main conversation turns.

Anthropic server tool web_search_20250305

Anthropic's native web search uses {"type": "web_search_20250305", "name": "web_search"}. Claude Code's built-in WebSearch sends this shape together with tool_choice: {type: "tool", name: "web_search"} to force a tool call. Because a forced tool_choice would cause the generic loop to re-emit the tool call on every round, the router uses a dedicated single-shot emulation path:

  1. Runs one backend turn to extract the query from the model's tool call (with a fallback that strips the Claude Code prefix "Perform a web search for the query: " from the user prompt if the backend does not emit a tool call).
  2. Calls the configured search provider once (no loop).

  3. Assembles the response in Anthropic's native format:

    • a server_tool_use block carrying the query, and
    • a web_search_tool_result block whose content field is either:
      • an array of ranked result objects (including an empty array [] for zero-hit searches), or
      • a single {"type": "web_search_tool_result_error", "error_code": "..."} object when the search provider fails (missing API key, HTTP 4xx/5xx, timeout, or parse error). The error_code is "too_many_requests" for HTTP 429 and "unavailable" for all other failures.

    Each result object's snippet is base64-encoded and stored in encrypted_content.

  4. Streams the matching SSE sequence (message_start, content_block_start/delta/stop for both blocks, message_delta, message_stop), matching the events Claude Code expects from api.anthropic.com.

The commercial native-Anthropic path (forward_to_anthropic_native) is untouched: when routing to a native Anthropic backend, the server tool is proxied verbatim for Anthropic to execute.

Gemini Native Google Search Grounding

When a request targets a type: gemini backend, Continuum Router supports Google's native Search Grounding without requiring a router-side web_search provider:

  • Chat Completions (POST /v1/chat/completions): Passing tools: [{"type": "google_search"}] (or {"type": "web_search"}, {"google_search": {}}, or web_search_options) automatically switches the request to the Gemini Native generateContent / streamGenerateContent endpoint with tools: [{"google_search": {}}]. Grounding citations returned in candidates[0].groundingMetadata are normalized into OpenAI-compatible choices[0].message.annotations (type: "url_citation") alongside the raw grounding_metadata object.
  • Responses API (POST /v1/responses): Passing tools: [{"type": "web_search"}] maps to native google_search grounding and populates output[].content[].annotations with url_citation entries.
  • Image Generation & Editing (POST /v1/images/generations, POST /v1/images/edits): Passing "google_search": true with nano-banana-pro (gemini-3-pro-image-preview) or nano-banana-2 (gemini-3.1-flash-image-preview) enables search-grounded image generation and preserves grounding_metadata on the image response (nano-banana-2-lite and nano-banana reject google_search: true with HTTP 400).

See Gemini Dual-API Routing and Exclusive Features for configuration details.

Configuration

Add a top-level web_search section to config.yaml:

web_search:
  enabled: true
  provider: serper              # serper | exa | brave | searxng
  api_key: "${SERPER_API_KEY}"  # env var substitution supported (not used by searxng)
  # base_url: "https://mirror.internal/serper/search"  # optional endpoint override (http(s), validated at load); required for searxng
  timeout_ms: 5000              # per-request timeout (100 - 60 000 ms)
  max_results: 5                # 1 - 20
  max_tool_iterations: 5        # hard cap on tool rounds (1 - 20)
  inject_policy: auto           # auto | always | never
  tool_name: web_search         # function name advertised to the model
  result_char_cap: 4000         # per-result snippet truncation cap (bytes)
  streaming_enabled: false      # mid-stream tool execution (advanced)
  loop_wall_clock_ms: 60000     # total wall-clock budget for the loop
  max_total_result_bytes: 32768 # combined byte cap on tool-result content

Custom provider endpoint

By default, each hosted provider calls its public endpoint:

  • Serper: https://google.serper.dev/search
  • Exa: https://api.exa.ai/search
  • Brave: https://api.search.brave.com/res/v1/web/search

To route searches through an internal mirror in air-gapped or policy-restricted deployments, set base_url to a valid http(s) URL:

  • Full endpoint URL: base_url must include the provider's path and is validated at load and hot-reload. Omit it to keep the public default unchanged.
  • Provider-specific URL rules:
    • brave and searxng: base_url must contain the endpoint path only (no query string or fragment), because the router appends ?q=&count= (Brave) or ?q=&format=json (SearXNG) on every request.
    • serper and exa: the router uses base_url verbatim as the HTTP POST target.

Self-hosted SearXNG

SearXNG is a self-hosted metasearch engine. Setting provider: searxng differs from hosted providers in two ways:

  • base_url is required: there is no public default endpoint. Set base_url to your instance's /search URL (omitting it fails config loading).
  • api_key is ignored: SearXNG uses no API key, and the router sends no authentication credential.
web_search:
  enabled: true
  provider: searxng
  base_url: "http://searxng.internal:8080/search"
  max_results: 5

The router calls GET <base_url>?q=<query>&format=json and maps each entry of the JSON results array (url, title, content) to a search result. Because SearXNG ships with only html enabled by default, you must enable json in the instance's settings.yml:

search:
  formats:
    - html
    - json

Two instance-side settings can produce errors that look like router faults:

  • HTTP 403 on every search: json is missing from search.formats in settings.yml. SearXNG returns 403 when a requested output format is disabled; no API key or token is involved.
  • HTTP 429 on every search: server.limiter: true classifies the router's non-browser requests (no HTML Accept header, no gzip) as bot traffic. Disable the limiter for a private instance, or add the router's IP to botdetection.ip_lists.pass_ip in limiter.toml. The Anthropic emulation path maps HTTP 429 to too_many_requests.

Additional runtime behaviors for SearXNG:

  • Result count truncation: SearXNG has no per-request result count parameter (each page returns whatever the aggregated engines produce, typically 20-40 hits), so the router truncates the ranked list to max_results.
  • Unresponsive engine warnings: an HTTP 200 with zero results is treated as a valid empty search. However, if the response also lists unresponsive_engines, the router logs those engine names at warn (searxng returned no results and reported unresponsive engines) so operators can spot instances whose upstream engines cannot reach the internet.
  • Instance-managed defaults: engine selection, language, and safe-search settings come from the SearXNG instance's settings.yml and are not passed per request.

Injection policies

inject_policy controls when the router adds the web_search tool definition to an outbound request:

  • auto (default): inject when the client sends enable_web_search: true or already references a web_search tool by name.
  • always: inject on every self-hosted request (while web_search is enabled).
  • never: never inject, allowing operators to pause automatic injection (for example, during A/B tests) without disabling the feature globally.

Per-backend overrides

web_search:
  enabled: true
  # ... global settings ...
  per_backend:
    sensitive-vllm:
      enabled: false           # disable for this backend only
    ollama-external:
      inject_policy: always
    # canary-vllm:
    #   enabled: true          # can opt in this backend even when the
    #                          # global enabled flag is false

Hub-delivered per-user credentials

When connected to Continuum Hub with control_plane.policy.enabled (control-plane builds), the router advertises search_credentials_v1 and receives per-user search credentials in the Hub policy envelope.

Credential precedence and gating

For requests authenticated by a Hub-issued API key:

  • User credential present: when the key owner (KeyEntry.user_id) has a Hub-delivered credential, that credential's provider and key serve the request instead of web_search.api_key. Because the provider is selected per credential, a single router can serve users across different search providers. base_url applies only when the user credential matches the locally configured provider.
  • No user credential: the router falls back to the deployment credential in web_search. Once the Hub has delivered a credential set, if a user has no personal credential and the deployment cannot authenticate on its own (a key-requiring provider with neither a valid api_key nor a base_url mirror), the router does not inject the tool.
  • Tier gate: when the key's tier sets web_search_enabled: false, the tool is never injected regardless of available credentials.

Local configuration and capability lifecycle

  • Operator opt-in: a Hub credential substitutes only the provider and key, never the injection policy. If the local web_search section is absent or disabled, the router never injects the tool.
  • Fallback to local config: requests that do not match a Hub key, or routers whose Hub sends no credentials, use the local web_search configuration.
  • Capability advertisement: the router advertises search_credentials_v1 at startup and after every config reload while a web_search section exists. Removing the web_search section on reload withdraws the capability and immediately drops all delivered keys and the acknowledged digest.

In-memory storage and envelope validation

  • Memory-only retention: delivered keys are held exclusively in memory. They are never written to disk (including the key-table sidecar), never logged, and always redacted in Debug output.
  • Envelope update semantics: an omitted field keeps the previous credential set, a present set replaces it, and an empty set revokes all delivered credentials.
  • Whole-set validation: the router rejects a credential update as a whole (keeping the previous set in force) if it contains more than 1,000 entries, duplicate user IDs, a provider other than serper, brave, or exa, or a digest mismatch.

Usage reporting and headers

  • Usage record fields: each usage record sent to the Hub includes:
    • web_search_queries: total provider calls made for the request, including failed calls.
    • web_search_provider: serper, brave, or exa (omitted for searxng).
    • web_search_credential_source: user or deployment.
  • Edge-case accounting:
    • Anthropic web_search_20250305 emulation records its query-extraction turn together with the search.
    • If a later backend turn in a tool loop fails, searches already executed are still reported in a zero-token usage record.
    • Across fallback chain attempts, each executed search is reported in at most one usage record.
    • An emulation turn whose backend response omits a usage block produces no usage record.
  • No per-request opt-out header: the router does not accept an x-continuum-websearch header. Injection is governed by the tier setting, per-backend inject_policy, and the client's tools list.

Hot reload

All web_search fields, including api_key, are re-read on configuration reload. You can rotate API keys or toggle the feature on and off without restarting the router.

Supported providers

Provider Status Endpoint
Serper Implemented https://google.serper.dev/search
Brave Implemented https://api.search.brave.com/res/v1/web/search
Exa Implemented https://api.exa.ai/search
SearXNG Implemented self-hosted; base_url required, no API key

All four providers implement the shared SearchProvider trait. Switching providers requires only changing provider: in the web_search section (plus setting base_url for SearXNG).

Security

  • Secret handling: API keys are expanded from ${ENV} references at load time and are never written to logs or HTTP error responses.
  • Debug redaction: Debug output for WebSearchConfig, SerperProvider, BraveProvider, and ExaProvider redacts api_key. SearxngProvider holds no credential and logs its configured base_url for operator visibility.
  • Content sanitization: search-result titles and snippets are stripped of HTML and control characters and truncated to result_char_cap before being appended to the model context.

Observability

When compiled with the metrics feature, the router exports these Prometheus metrics:

  • web_search_calls_total{provider, outcome}: counter incremented once per executed search call.
  • web_search_call_duration_seconds{provider, outcome}: latency histogram of search provider calls.
  • web_search_iteration_cap_total{component="loop"}: counter incremented whenever a request hits max_tool_iterations.
  • web_search_injections_total{backend_type}: counter incremented once per request where the router injected the web_search tool.

Safety bounds

The non-streaming tool-execution loop enforces four independent guards:

  • Iteration cap (max_tool_iterations): caps the number of backend round-trips (default 5).
  • Wall-clock budget (loop_wall_clock_ms): caps total elapsed time spent in the loop (default 60000 ms). When exceeded, the router returns the model's last response unchanged without surfacing an error to the client.
  • Payload budget (max_total_result_bytes): caps the cumulative byte length of tool-result content appended across all iterations (default 32768 bytes). Once exceeded, subsequent tool results are replaced with a short "tool-result budget exhausted" error message so the model can close the tool-call cycle and answer.
  • Orphan tool-call guard: tool calls whose function.name does not match the configured tool_name receive a structured tool-role error message rather than being executed, preventing the backend from rejecting the next turn due to a dangling tool_call_id.

Model aliasing (Claude Code compatibility)

Clients such as Claude Code hard-code specific Anthropic model names for internal helper tasks. For example, WebFetch content summarization and WebSearch query extraction call claude-haiku-4-5-20251001. On a router serving only self-hosted models, those internal calls would fail with ModelNotFound unless aliased.

The top-level model_aliases section rewrites the incoming model field before backend selection, following the ANTHROPIC_DEFAULT_{HAIKU,SONNET,OPUS}_MODEL convention popularized by cc-switch:

model_aliases:
  haiku: GLM-5_1          # any incoming name containing "haiku"
  sonnet: GLM-5_1         # any incoming name containing "sonnet"
  opus: GLM-5_1           # any incoming name containing "opus"
  reasoning: GLM-5_1      # used when thinking.type is enabled | adaptive
  default: GLM-5_1        # catch-all; leave unset to disable
  exact:                  # full-name pins, take precedence
    claude-haiku-4-5-20251001: GLM-5_1

Key aliasing rules:

  • Match order: exact -> reasoning (when thinking is enabled) -> haiku -> opus -> sonnet -> default.
  • Substring matching: tier keys (haiku, opus, sonnet) match via case-insensitive substring check on the incoming model name. Every rewrite is logged at info level.
  • Endpoint scope: model_aliases applies only to /anthropic/v1/messages; /v1/chat/completions is unaffected. If you route to a native Anthropic backend and do not want model names rewritten, leave model_aliases unset.

Caveats

  • Streaming support: the non-streaming tool loop and the Anthropic web_search_20250305 server-tool emulation path (including streaming SSE) are enabled by default. Mid-stream tool execution for the generic custom-tool loop is gated behind streaming_enabled: false by default; when disabled, streaming requests on the generic path pass through the upstream SSE stream without router-side tool execution.
  • Malformed tool arguments: if a model emits malformed JSON instead of {"query": "..."}, the router returns a tool-role error message to the model rather than failing the request with an HTTP error.
  • OpenAI function-calling schema: the injected tool uses OpenAI's tools[].function.parameters schema so self-hosted models trained on OpenAI-style tool calling behave predictably.
  • Non-cryptographic encrypted_content: in the emulated web_search_tool_result block, encrypted_content contains a base64-encoded snippet rather than a cryptographically signed blob. Because the router acts as the server across turns, an opaque base64 payload is sufficient for Claude Code round-trips.

Web search history on conversion backends

Claude Code resends the server_tool_use and web_search_tool_result blocks of an earlier search on every later turn. A conversion backend (OpenAI-compatible, Gemini, or a responses_only model) receives them as text in the same message and position: Web search: <query>, then Web search results: with one - <title> (<url>): <snippet> line per result (the snippet is encrypted_content decoded from base64, omitted when it does not decode), Web search results: none for an empty list, or Web search failed: <error_code>. A malformed block of either type, and server_tool_use for any other server tool, is still rejected with 400. Native Anthropic and Bedrock backends receive the blocks unchanged.