Vector Stores Passthrough: Design Record¶

This note records the design for the /v1/vector_stores passthrough (issue #2412). The routes are implemented in src/http/handlers/vector_stores.rs and src/services/vector_stores.rs; the buffered forwarding they share with the fine-tuning passthrough lives in src/services/json_passthrough.rs.

Implemented surface¶

Each response is the backend's own status, content type, and body. A list body is the exception to byte-identical relaying: its data array keeps only the caller's own stores (see Store bindings).

Policy layers¶

A vector-store call is not a chat dispatch. The router applies the API route group's auth, API-key scope, global, per-client, and per-key rate limits to every call, charges the per-backend dispatch limit on create and list, and deliberately does not run the layers that assume a completion payload:

• Request-parameter policy: not applied. It rewrites completion sampling and tool fields that a store payload does not carry.
• Guardrails: not applied. The input and output gates judge message and completion text; the router never sees the indexed content.
• Response caching: not applied. A cached listing, create, or search would hide state changes.
• Per-model generation budget: not charged. A store call generates no tokens.
• Metering: applied as a non-completion request. The router records request count, latency, and key and backend attribution with zero tokens (token_source: none), because the answer has no usage block to count.

The JSON-bearing routes carry an explicit 1 MiB body bound. A create body holds a name, metadata, and an expiration policy; an attachment body holds a provider file id and a chunking strategy; a search body holds a query and filters. The bound keeps the router from buffering an arbitrary payload before forwarding it.

Design¶

• Backend scope: OpenAI-wire types on http/https only (openai and generic). Azure OpenAI is excluded because its Vector Stores endpoint is a different path with a required api-version. A backend that does not serve the API is excluded from the candidate set before selection, and a request that no capable backend can serve answers a structured 400 vector_stores_not_supported, not a router-level 404.
• Backend selection: a store call names no model, so the client's x-backend header is not consulted. Create selects one capable backend with the configured selection strategy over every candidate the caller may reach; list does the same. A per-request pin on a lifecycle call could send it to a backend that does not own the store, so the binding is authoritative once a store exists.
• Shared forwarding: the fine-tuning passthrough already buffers a JSON request and relays a backend answer. Both surfaces now use json_passthrough::forward for the credential, organization id, request extensions, resolved user profile, router hop count, read bound, and circuit settlement; each keeps its own selection rules, binding table, and policy decisions.

Store bindings¶

A create answer is buffered long enough to read the store id. The router then records (principal, store id) -> backend in a per-AppState LRU table, the same shape as the fine-tuning job table and the Responses passthrough table (services::responses::PassthroughRoutes, issue #1941). Every store-scoped call looks up that binding and re-checks it against the live configuration and the caller's allowed_backends; an unknown or foreign id answers 404 vector_store_not_found and nothing is sent upstream.

GET /v1/vector_stores applies the same ownership rule. The provider's list carries every store on the backend account, including stores other principals created through this router on the same credential, so the router keeps only the entries whose id it bound to this caller on the backend that served the page. A store this router did not bind (created before a restart, or through another router process) is therefore not listed, which matches retrieve, modify, and delete answering not-found for the same store. The page's has_more, first_id, and last_id describe the backend page rather than the filtered list, so a caller pages until has_more is false.

The binding table is process memory. A store created before a router restart still exists on the backend, but this router no longer knows which backend to call, so it answers not-found. The table is one 10,000-entry LRU shared by every principal, so a principal that creates more stores than the table can hold can evict another principal's binding. That store remains on the provider, but this router answers not-found for it and hides it from a list; the impact is availability, not disclosure. Per-principal quotas are a follow-up.

Attached files¶

An attachment body names a provider file id. The router's own /v1/files stores uploads on local disk (files.storage_path) and resolves them into inference requests; it does not upload them to a provider. A router file id therefore is not a provider file id, and this design adds no provider upload or translation step. The same boundary applies to a fine-tuning job's training_file.

The store binding gates the attachment: a caller can only attach to, list from, or remove from a store it created. The provider file id itself is opaque, so the router does not track which principal uploaded it. On a provider credential shared by several tenants, a leaked provider file id can therefore be attached to another tenant's store and read back through search; the API page states this so operators do not share one credential across tenants that must not see each other's files.

Remaining open questions¶

• Binding persistence: how the binding table is stored and recovered across restarts. The in-memory table enforces ownership for the process lifetime only.
• Multi-backend listing: whether GET /v1/vector_stores should merge pages from every capable backend instead of selecting one. Merging would invent a normalized store schema, which the issue asks the router not to do.
• Provider file ownership: whether the router should map its own file ids to provider uploads, or track which principal owns each provider file id. Neither is part of this passthrough.