Rate Limiting¶

Continuum Router rate-limits requests to prevent abuse, allocate resources fairly, and protect backends from overload. The rate limiting system uses a token bucket algorithm with multiple layers of protection.

Overview¶

The router implements multi-tier rate limiting:

• Per-client limits: Prevent individual clients from overwhelming the system
• Per-backend limits: Keep a backend at its configured capacity by routing around it, and refuse only when every backend serving the model is at its limit (off by default; see Per-Backend Limits)
• Per-API-key limits: Apply limits per authenticated key, with per-key overrides
• Per-user and per-team limits: Key limits on the caller's user or team identity so several keys of one user, or every member of one team, share one allowance
• Per-model limits: Protect specific (expensive) models
• Global limits: Ensure overall system stability
• Token budgets: Bound tokens per minute rather than request count, at admission and again at completion (off by default; see Token Budgets)

Where limits apply¶

Each rate-limit dimension is mounted at the outermost layer where its key identity is available, and each dimension is enforced at a single point:

• Global (limits.global): Mounted once across the entire application, outside authentication and CORS. Every request is charged once, including API requests later rejected with 401 (so API-key guessing is metered), admin requests (including failed admin authentication), /status, /version, WebUI, Files, metrics, CORS preflights, and the 404 fallback.
• Per-client (limits.per_client): Mounted on API routes (/v1/*, /anthropic/*, /embed_sparse) outside authentication. Because it keys on the connection peer address before authentication runs, it contains unauthenticated floods from a single source IP before they can drain the shared global bucket.
• Identity dimensions (per_api_key, per_user, per_team, per_model): Mounted on API routes inside authentication, where AuthContext is available. This layer never touches the global or per-client buckets.
• Per-backend (limits.per_backend): Applied during backend selection rather than at request admission, because no backend is assigned when a request first arrives. The selector filters out backends with empty buckets and routes to another backend serving the same model, charging the chosen backend's bucket once upon dispatch. A request is refused only when every candidate backend is exhausted. See Per-Backend Limits.

On API routes, evaluation proceeds in order: global -> per-client -> authentication -> per-key -> per-user -> per-team -> per-model -> handler -> selection (per-backend) -> dispatch. Because each mount enforces disjoint dimensions and per-backend charging happens once per dispatch, a request is never double-counted against any bucket.

Exemptions and unauthenticated traffic:

• Health probes: The global mount unconditionally exempts /health, /health/ready, and /healthz. These handlers read only bounded in-memory state, ensuring a saturated router still answers orchestrator liveness and readiness probes.
• Metrics endpoint: Exempt only when metrics.enabled is true, following the configured metrics.path (if metrics are disabled, /metrics is metered like any other unmatched path). Because a scrape rebuilds backend gauges and serializes the full registry, scrape cost scales with metric cardinality and is not bounded by the rate limiter. Always protect the metrics endpoint with metrics.auth and network policy.
• Unauthenticated and unmatched traffic: Requests rejected with 401 and unknown paths hitting the 404 fallback draw from limits.global. On API paths, per_client isolates single-IP floods in their own bucket first, but non-API paths rely solely on limits.global. Size limits.global with headroom above your expected API rate, or drop unwanted traffic at an upstream proxy.

Header attribution and refusal precedence¶

Because multiple rate-limit layers evaluate the same request, response headers follow strict attribution rules:

• On admitted requests (2xx): x-ratelimit-* headers report only quota tied to an authenticated identity (per_api_key). If the per_api_key dimension did not evaluate the request, no x-ratelimit-* headers are returned.
• Shared buckets are hidden on success: limits.global, per_model, per_backend, per_user, and per_team reflect capacity shared by more than one caller, while per_client is shared by every caller behind the same source IP (such as a NAT or corporate proxy) and runs before authentication. All of these dimensions enforce as configured, but their levels are not advertised on admitted responses. Configure per_api_key if clients need quota headers on successful calls.
• On refused requests (429): The dimension that rejected the request determines error.details.limit_type, retry-after, and x-ratelimit-* (even for shared buckets, so clients receive a retry hint). A token-budget refusal by a shared dimension is the exception: it withholds that bucket's level, as described under Token Budgets. If both global and an inner dimension would refuse the request, the client receives the global refusal because the global mount runs first. Retry-After is always at least 1.
• CORS visibility: When CORS is enabled, retry-after and x-ratelimit-* are added to Access-Control-Expose-Headers so browser scripts can read retry hints. The global mount (which sits outside the CORS layer to meter preflights) also attaches Access-Control-Allow-Origin directly to short-circuited 429 responses.
• Outer-layer token consumption: A request refused by an inner limit (such as per_api_key) still consumes one token from limits.global and one from limits.per_client, as those outer mounts evaluated the request before it reached the inner layer.

What the headers do and do not reveal¶

x-ratelimit-* headers on successful responses reflect only the caller's own authenticated per_api_key bucket. Any response that did not undergo per_api_key evaluation omits quota headers altogether, including:

• 401 authentication failures and 404 fallback responses
• Unauthenticated 200 endpoints such as /version
• Requests matching IP whitelist or bypass_keys
• Unauthenticated requests on API paths

This uniformity prevents side-channel key probing. Because per_client runs before authentication, reporting per_client headers on 401 responses would distinguish a bypass_keys holder (exempt, no headers) from an arbitrary invalid token (metered, headers present), allowing an attacker to test suspected bypass keys at one request per guess. Withholding per_client headers eliminates this signal.

In permissive authentication mode with per_api_key configured, recognized keys receive per_api_key headers while unrecognized tokens do not. Because permissive mode is a development and trusted-network convenience that already grants identical model and backend access to valid and invalid keys alike, this distinction does not cross a security boundary. Deployments where key validity is sensitive should set api_keys.mode: blocking so unrecognized keys are rejected with 401 before any identity dimension runs.

Reading quota without spending it¶

GET /v1/quota reports the caller's own limits and remaining values without charging an identity-bound bucket, so a client can poll it without consuming what it reads. A key already at its per_api_key bucket or its max_tokens_budget still gets 200 with remaining: 0.

The document includes only values tied to the authenticated identity:

• request_rate and token_rate: the per-key request and minute-token buckets.
• user_request_rate and team_request_rate: the per-user and per-team request buckets, when configured. These are shared by that user's or team's keys and by no other identity.
• inflight: the current count against max_in_flight_requests.
• token_budget: the long-window max_tokens_budget minus recorded usage and in-flight reservations.
• hub_*: the Hub tier windows, when the token matches a synced key.

Shared fleet levels (global, per_client, per_model, per_backend), the batch dimensions, and every key id, name, user, and team are never included. A quota response carries no x-ratelimit-* or x-continuum-ratelimit-* header.

The shared global and per_client margins are still charged at the /v1/models cost of 0.1, and either can refuse the read with its normal 429. This keeps the route metered before authentication like every other API path, while their levels stay off the wire.

Unauthenticated callers: in permissive mode a request with no token or an unrecognized token receives 200 with an empty dimensions list, byte-identical to a recognized key with no configured limit. In blocking mode the normal 401 and 403 apply.

Because the long-window budget is also reported, the permissive-mode distinction covers max_tokens_budget as well as per_api_key: a key with either configured produces a document an unrecognized token does not. Deployments where key validity is sensitive should set api_keys.mode: blocking.

Store outages: with rate_limiting.redis.fallback_to_memory: true, an unreachable Redis reads this replica's bucket, reports scope: "replica" on the affected rows, sets degraded: true, and returns 200. With fallback_to_memory: false, the read receives the existing 503 with Retry-After: 5. When the Hub shared minute store is unreachable, the Hub rows come from this replica's local windows with degraded: true, matching the fail-open rule for those limits.

Table of Contents¶

• Configuration
• Client Identification
• Rate Limiting Strategies
• Response Headers
• Monitoring
• Bypass Mechanisms
• Storage Backends
• Best Practices

Configuration¶

Basic Configuration¶

rate_limiting:
  enabled: true
  storage: memory  # or "redis" for distributed setups

  limits:
    per_client:
      requests_per_second: 10
      burst_capacity: 20
    per_backend:                 # opt-in: a routing filter, not a refusal; see Per-Backend Limits
      requests_per_second: 100
      burst_capacity: 200
    global:
      requests_per_second: 1000
      burst_capacity: 2000

Per-API-Key Rate Limiting¶

Three mechanisms apply to API keys, in this precedence order:

1. A per_api_key dimension inside rate_limiting.limits that acts as the default for every recognized key that has no per-key override of its own:

   rate_limiting:
     limits:
       per_api_key:
         requests_per_second: 10
         burst_capacity: 20
   

2. A per-key rate_limit override (requests per minute) on individual entries in the api_keys section. When set, it replaces the per_api_key default for that key; rate_limit: 0 disables per-key limiting for that key entirely:

   api_keys:
     api_keys:
       - key: "premium-user-key"
         name: "Premium User"
         rate_limit: 600        # requests per minute
   
       - key: "standard-user-key"
         name: "Standard User"
         rate_limit: 60         # requests per minute
   

3. Two optional per-key values refine the rate in effect for that key:

   • rate_limit_burst is the bucket capacity for whichever request rate applies (the key's own rate_limit when set, otherwise per_api_key). Unset keeps the derived capacity of twice the per-second rate.
   • rate_limit_tokens_per_minute replaces the per_api_key minute token budget for this key. 0 disables the key's minute budget, and unset keeps the configured dimension budget. The other admission-time dimensions (global, per_client, per_model) still charge the key's requests, and a per-key day budget is not part of this override.

   api_keys:
     api_keys:
       - key: "gateway-key"
         rate_limit: 6000               # requests per minute
         rate_limit_burst: 500          # requests the key may spend at once
         rate_limit_tokens_per_minute: 2000000
   

Per-key limits bind only to recognized keys: the dimension reads the identity that authentication attaches. Anonymous callers (no key, or an unknown key) are never per-key limited, in either auth mode. They are still bounded by per_client and global, which are mounted outside authentication and so apply whether or not the request is later rejected with 401.

The request-count per_api_key bucket is keyed on a deployment-stable, domain-separated hash of the credential, so with storage: redis every replica charges one shared bucket. The derivation carries no per-process salt; the hash is only ever a bucket key and is never sent to a client. Upgrading to this derivation re-keys the existing request-count per_api_key buckets once, and the old Redis keys age out at the configured TTL (issue #2899).

Override scope¶

The per-key surface covers one key's request rate, its burst, and its minute token budget. These limits stay outside it:

• A budget shared by all of one user's keys: tracked separately in issue #2381.
• Per-user and per-team dimensions, in-flight caps, and fair queuing (tier overrides are configured separately under rate_limiting.tier_overrides).
• A per-key day token budget: rate_limit_tokens_per_minute changes only the minute bucket.

Per-User and Per-Team Rate Limiting¶

per_user and per_team key their buckets on the identity the API key already resolves, not on the credential. Two keys that belong to one user therefore draw on one bucket, and every member of a team draws on the team bucket.

• Identity source: per_user uses the key's user id. per_team uses the key's organization id. For a Hub-synced key, these are the user_id and team_id in the synced entry; for a static key, they are the user_id and organization_id fields in api_keys.
• Unresolved identity: A request whose key carries no user id, or no team id, skips the configured dimension. This covers anonymous callers, static keys configured without an identity, and a Hub key with no synced team_id (the continuum-hub placeholder is not a team identity). The skip is counted on rate_limit_skipped_total{dimension="user"} or rate_limit_skipped_total{dimension="team"}. Such a request is never charged to a bucket shared with other identity-less keys.
• Refusals and metrics: A refusal names the dimension in error.details.limit_type (user or team) and is counted on rate_limit_exceeded_total{limit_type="user"} or rate_limit_exceeded_total{limit_type="team"}. Both dimensions are shared by more than one credential, so their bucket levels are withheld from response headers.
• Storage and reload: The in-memory bucket maps and the Redis key space (per_user:<hash>, per_team:<hash>) carry a replica-stable hash of the identity, so every replica shares one bucket. A configuration reload applies the new settings immediately and resets the tenant buckets, like the other dimensions.
• Token budgets: tokens_per_minute and tokens_per_day work on these dimensions with the same reservation lifecycle as the others. A per-user budget is shared by all of that user's keys, and a per-team budget by every member of the team. A refusal reports token_dimension: "user" or token_dimension: "team" and withholds the bucket level.

rate_limiting:
  limits:
    per_user:
      requests_per_second: 20
      burst_capacity: 40
      tokens_per_minute: 200000   # Shared by all keys of the user
    per_team:
      requests_per_second: 100
      burst_capacity: 200
      tokens_per_day: 50000000    # Shared by every member of the team

Concurrent In-Flight Caps¶

max_in_flight_requests bounds how many requests of one tenant are resident at once, which a request rate cannot express: ten requests per second of thirty-second generations keeps roughly three hundred streams open. Set it beside the other settings on per_api_key, per_user, per_team, and per_client, and leave it at 0 (the default) to keep that dimension concurrency-unlimited.

• Admission: the slot is taken after every request-count and token-budget check passes, so a refused request takes none. A request over the cap is refused immediately with 429 and never queued.
• Refusal shape: limit_type: "inflight", error.details.inflight_dimension naming api_key, user, team, or client, and Retry-After: 1. The router does not estimate when a stream will end, so it does not advertise a longer wait.
• Release: the slot is an RAII guard attached to the response body. A stream holds it until its last frame, a handler failure releases it, and a client disconnect releases it with the request future.
• Ignored dimensions: global, per_backend, and per_model. Use server.max_concurrent_requests for a router-wide residency bound; config validate warns when the field is set on any of the three.
• Accounting: per replica, like the in-memory request buckets, so a fleet of N replicas enforces up to N times the configured cap. Redis does not share the counts.
• Bypass: a full whitelist or bypass_keys exemption skips the cap, and a scoped_bypass_keys entry naming inflight skips only this class.
• Metrics: rate_limit_inflight_requests{dimension, tenant} reports current holds and is removed at zero, rate_limit_inflight_refusals_total{dimension, tenant} counts refusals, and a refusal also counts on rate_limit_exceeded_total{limit_type="inflight"}.

rate_limiting:
  enabled: true
  limits:
    per_user:
      requests_per_second: 20
      burst_capacity: 40
      max_in_flight_requests: 64    # Concurrent streams across the user's keys
    per_team:
      requests_per_second: 100
      burst_capacity: 200
      max_in_flight_requests: 512   # Concurrent streams across the team

Per-Tier Rate Limiting¶

A Hub tier can carry its own per-key limits, so a paying tier gets a larger rate and burst than a free tier without changing limits.per_api_key for every key:

rate_limiting:
  limits:
    per_api_key:
      requests_per_second: 10
      burst_capacity: 20
      tokens_per_minute: 100000
  tier_overrides:
    tier_premium:
      requests_per_second: 50
      burst_capacity: 100
      tokens_per_minute: 500000
    tier_free:
      requests_per_second: 2
      burst_capacity: 4

• Resolution: the limiter matches the presented credential against the Hub-synced key table and reads the key's tier_id, the same identity smart_routing.routing_policies[].when.key_tier names. This needs the control-plane build feature, control_plane.enabled: true, policy enforcement, and a synced policy. Without them the map is inert, every key keeps the settings it would have without it, and config validate and the startup log warn.
• What an entry replaces: requests_per_second and burst_capacity always apply to keys of that tier. tokens_per_minute replaces the per_api_key minute budget only when written; omit it to inherit the dimension's budget. The day budget is not part of the override and stays as configured.
• Precedence: an explicit api_keys[].rate_limit (with its rate_limit_burst and rate_limit_tokens_per_minute) is the most specific setting and wins over the tier. The tier wins over limits.per_api_key.
• Buckets and headers: the resolved tier is part of the request-rate and minute-budget bucket keys, so a key the Hub moves to another tier starts on that tier's bucket instead of reusing the old capacity. The x-ratelimit-* headers report the limits actually applied to the request.
• Composition with Hub minute limits: tier_overrides sets local limits only. The Hub tier's rpm, input_tpm, and output_tpm windows, including their shared counters under control_plane.policy.minute_limits, are enforced separately by the control-plane path. A local override neither replaces nor widens them: a request is admitted only when both the local limiter and the Hub windows admit it.
• Validation: requests_per_second and burst_capacity are required and must be greater than zero, burst_capacity must be at least requests_per_second, and an unknown field inside an entry is a load error. Each tier id must be non-empty and at most 128 bytes.

Bypass Configuration¶

Certain clients can bypass rate limiting entirely:

rate_limiting:
  # Whitelist IPs that bypass rate limiting
  whitelist:
    - "192.168.1.0/24"
    - "10.0.0.1"

  # API keys that bypass every rate-limit dimension
  bypass_keys:
    - "admin-key-123"
    - "monitoring-key-456"

bypass_keys and whitelist exempt a caller from every dimension. To exempt one class of dimension instead, add a scoped_bypass_keys entry:

rate_limiting:
  scoped_bypass_keys:
    # A trusted gateway: no request-count limits, token budgets still enforced.
    - key: "gateway-key-789"
      dimensions: [request_rate]

    # A batch importer: token rate limits waived, request count still enforced.
    - key: "batch-key-012"
      dimensions: [tokens_per_minute, tokens_per_day]

The dimensions list accepts these values:

• request_rate: the request-count buckets global, per_client, per_api_key, and per_model.
• tokens_per_minute: the minute token budget on every admission-time dimension.
• tokens_per_day: the day token budget on every admission-time dimension.

An entry exempts only the classes it names, so the dimensions it does not name keep charging the key. A full bypass_keys entry for the same key still exempts every dimension. Each matching request is counted on rate_limit_bypassed_requests_total, and the key value is masked in configuration exports.

Client Identification¶

The router identifies clients using the following priority order:

1. API Key from the Authorization: Bearer <token> or x-api-key header (preferred)

   • Provides accurate tracking across different IPs

2. X-Forwarded-For header (proxy/load balancer scenarios)

   • Honored only when the request comes from a trusted proxy
   • With use_rightmost_xff: true (default), the rightmost IP in the chain is used, which is harder to spoof

3. X-Real-IP header (alternative proxy header)

   • Also honored only from trusted proxies

4. Direct IP address (when no proxy headers present)

   • Used when the request comes directly to the router

Trusted Proxy Configuration¶

rate_limiting:
  # Proxies allowed to set X-Forwarded-For / X-Real-IP headers
  trusted_proxies:
    - "10.0.0.0/8"
    - "192.168.1.1"
  # Use the rightmost IP in the X-Forwarded-For chain (default: true)
  use_rightmost_xff: true

Forwarding headers from untrusted sources are ignored, so clients cannot evade per-client limits by spoofing X-Forwarded-For.

Unix Socket Listeners¶

A Unix domain socket peer has no IP address, so on unix: listeners the per-client dimension and the IP whitelist cannot bind and are skipped gracefully. Each skip is recorded in the rate_limit_skipped_total{dimension="client"} metric and a debug log. Global and per-API-key limits enforce normally on such listeners.

Rate Limiting Strategies¶

Token Bucket Algorithm¶

The router uses the token bucket algorithm, which allows for burst traffic while maintaining long-term rate limits:

• Bucket capacity: Maximum number of tokens (burst_capacity)
• Refill rate: Tokens added per second (requests_per_second)
• Token cost: Each request consumes one token

How It Works¶

1. Each client starts with a full bucket of tokens
2. Tokens are consumed with each request
3. Tokens refill at a constant rate
4. Requests are rejected when bucket is empty

Per-Backend Limits¶

limits.per_backend is a capacity constraint on each backend rather than a caller quota, so it acts as a routing filter instead of an admission-time refusal:

• Candidate filtering: When multiple backends serve the requested model, any backend with an empty per-backend bucket is excluded before the selection strategy runs. This filter runs alongside health and circuit-breaker filtering on every request path (chat, streaming, Responses, Anthropic Messages, images, realtime, ACP).
• Dispatch charging: A backend's bucket is charged one token only when the request is actually dispatched to it. Backends excluded by filters, passed over by the selection strategy, or left behind on a fallback hop are not charged; cache hits and guardrail-blocked requests are not charged either. Every dispatch costs one token regardless of route.
• Refusal only when all backends are exhausted: If every backend serving the model has an empty bucket, the router returns 429 with error.details.limit_type set to backend (distinct from the 503 service_unavailable returned during an outage). When a fallback chain is configured, this 429 triggers the chain (via the default error_codes trigger) so the request advances to a fallback model with available capacity.
• Disabled by default: Like per_api_key, per_backend is off unless explicitly configured.
• Interaction with bypasses and Redis:
  • whitelist and bypass_keys exempt callers from admission mounts, not backends from capacity limits. Exempt callers still route around exhausted backends and receive 429 only if every candidate backend is exhausted.
  • With storage: redis, per-backend buckets are shared across router replicas, and a Redis outage follows the store outage policy. Under redis.fallback_to_memory: false the request is refused with 503 service_unavailable and Retry-After: 5, not with the capacity 429, and a fallback chain is not triggered.

rate_limiting:
  limits:
    per_backend:
      requests_per_second: 50    # steady-state dispatches per second, per backend
      burst_capacity: 100        # dispatches a backend can absorb at once

Per-Model Limits¶

In addition to the per-client, per-backend, per-API-key, and global dimensions, limits can target individual models. This is useful for protecting expensive models while leaving cheaper ones unconstrained:

rate_limiting:
  limits:
    per_model:
      gpt-5.6-sol:
        requests_per_second: 2
        burst_capacity: 5
      gpt-5.6-luna:
        requests_per_second: 20
        burst_capacity: 50

Which name the bucket keys on¶

The bucket keys on the client-facing model name exactly as sent in the request, with no case folding, whitespace trimming, or normalization. A per_model key that does not match the exact string sent by clients will never be charged.

The router extracts the model name from the request path or body and ignores the X-Model header on every route. Because evaluation happens before handler routing, two rules apply:

• model_aliases: Alias resolution runs inside the handler after per_model evaluation. A limit on the alias name bounds requests using that alias, whereas a limit on the alias target bounds only requests that name the target directly. If an upstream model is reachable under multiple client-facing names, configure a per_model entry for each name.
• model_experiments: Variant assignment also runs after per_model evaluation. A limit on the experiment's client-facing model bounds the entire experiment across all arms, whereas a limit on a variant model bounds only requests that name that variant directly.

Which routes are covered¶

per_model applies only to routes that provide a model name in the path or JSON body:

• Request path: GET /v1/models/{model} (including namespaced IDs such as vendor/model).
• Top-level model field in the JSON body: POST /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/rerank, /embed_sparse, /v1/responses, /v1/responses/compact, /v1/images/generations, /v1/audio/speech, /anthropic/v1/messages, and its root-level alias /v1/messages. The admission layer inspects the body once within the standard 10 MB request-body limit and forwards it unchanged.

All other routes are bounded by global, per_client, and per_api_key instead:

• /v1/images/edits and /v1/images/variations (multipart form bodies parsed downstream)
• /anthropic/v1/messages/count_tokens and /v1/messages/count_tokens (token counting only)
• /tokenize, /detokenize, and their /v1 aliases (tokenizer-only calls)
• Routes without a model parameter: GET /v1/models, /v1/models/extended, /v1/models/proxy, /v1/batches, /v1/responses/{id}, and the realtime WebSocket endpoint

If a request on a covered route omits model, is not valid JSON, or exceeds the body inspection limit, per_model does not charge it. The request proceeds to the handler for normal validation and error reporting, and the router increments rate_limit_skipped_total{dimension="model"}.

Token Budgets¶

The request-count dimensions bound how many requests arrive, not how much work each one carries. One request with a 200k-token prompt, or a reasoning model running at its maximum thinking budget, stays under any request-rate cap while occupying a backend for minutes. Set tokens_per_minute, and optionally tokens_per_day, beside requests_per_second to bound tokens on the same dimensions:

rate_limiting:
  limits:
    per_api_key:
      requests_per_second: 10
      burst_capacity: 20
      tokens_per_minute: 100000
      tokens_per_day: 2000000
    global:
      requests_per_second: 1000
      burst_capacity: 2000
      tokens_per_minute: 5000000
      tokens_per_day: 50000000

• Admission-time dimensions only: per_api_key, per_user, per_team, per_client, per_model, and global each carry an independent token bucket. A request is charged on each configured dimension it has an identity for: the key identity, the key's user identity, the key's team identity, the peer address, the per_model entry matching the client-facing model name (other model entries do not apply), and the fleet-wide global bucket. per_backend is a selection-time filter charged once per dispatch, so a token budget has nowhere to attach there; a value set on it is ignored, and config validate warns about it.
• Reserved together, once: all configured token dimensions are reserved in one step at the identity layer, inner to authentication and before dispatch, so a request is charged exactly once per dimension. The request-count counterpart of global and per_client is still enforced at its own outer mount.
• One minute of burst: each bucket holds tokens_per_minute tokens and refills at tokens_per_minute / 60 per second, so a caller may spend its whole minute at once. 0, the default, leaves that dimension token-unlimited, and a config without the field behaves as before.
• Optional day budget: tokens_per_day adds a second, independent bucket beside the minute one on the same dimensions, charged by the same reservation and reconciliation. A request must fit both budgets. Its capacity is one day of the rate and it refills at tokens_per_day / 86400 per second, so a caller may spend its whole day at once and the bucket is whole again one day after the drawdown. It is a leaky-bucket window, not a calendar-day reset at midnight.
• Admission reserves an over-estimate: before dispatch, the router reserves the whole-request input estimate plus the output limit on every configured dimension that applies. The output limit is the largest of max_tokens, max_completion_tokens, and max_output_tokens, and defaults to 4096 when the request states none, so a request with no declared cap cannot hold only its small input. A float or numeric-string spelling such as 32000.0 or "32000" reads as the same ceiling, because pydantic-based upstreams coerce it and then generate up to it. A request larger than a whole bucket is refused immediately because waiting cannot make it fit.
• Size-aware refusals: a token-budget refusal's Retry-After reports the time the exhausted budget needs to refill the refused request's own cost, not a fixed window. A 10-token request and a 200k-token request refused by the same budget receive different hints. The hint is floored at 1 second and capped at 86400.
• Deferred admission (off by default): when rate_limiting.deferred_admission.enabled is set, a refused request whose wait fits max_wait_ms (default 2000) is held for that wait instead of refused with a 429. The hold is bounded per router process by max_held_requests (default 1000), a count each replica keeps for itself; a request past either bound gets the immediate 429 with the size-aware Retry-After. A held request has not been dispatched, so it holds no backend slot and no token reservation: the reservation is taken only once a wait succeeds. A request larger than the whole burst is never held, because waiting cannot make it fit.
• Fair queuing across tenants (off by default): when rate_limiting.fair_queuing.enabled is set, a request an identity dimension (per_api_key, per_user, per_team, per_model) refuses is held in its tenant's bounded queue instead of answered with a bare 429, and the queues are served by weighted fair scheduling, so one tenant's flood cannot starve another. The tenant is the same user, key, or team identity the limiter charges (tenant_dimension, default user) and weights tunes each tenant's share. Every level is bounded (max_queue_depth_per_tenant 32, max_total_queued 4096, max_tenants 1024, max_in_flight 8, max_wait_ms 5000), and a request that does not fit a bound, or whose wait cannot fit the budget, is refused with x-fair-queue-position and x-fair-queue-wait-ms. Queued time counts against the request's total timeout. The global and per-client refusals stay immediate because their mounts run before authentication, and deferred_admission still handles token-budget refusals, so both modes can be enabled together. New metrics are rate_limit_fair_queue_depth, rate_limit_fair_queue_tenants, rate_limit_fair_queue_wait_seconds{tenant_dimension}, rate_limit_fair_queue_grants_total{tenant_dimension}, and rate_limit_fair_queue_refused_total{tenant_dimension, reason}. See Fair Queuing Across Tenants.
• Completion reconciles against reported usage: unused reservation is refunded, an overage is charged, and a failure, a response with no usage block, or a client cancellation releases the reservation in full. A streamed reply settles the same way when the stream reaches its normal end, against the backend's usage block or the router's own count when the backend reported none; a stream cut before its end (a cancellation, a timeout, an upstream error, or a guardrail block) releases the reservation in full.
• Only text-generation routes are charged: /v1/chat/completions, /v1/completions, /v1/responses, /v1/responses/compact, /v1/messages, and /anthropic/v1/messages. Every other route is bounded by the request-count dimensions alone.

The input estimate walks every message, tools, functions, and response_format, and adds the prompt fields the chat walk does not see (prompt, input, instructions, system), so the completions, Responses, and Anthropic shapes are charged for their own input. A body the router cannot inspect costs the request nothing and is admitted unchanged; the route handler still owns validation and the error shape.

A dimension with no identity for the request is skipped and counted on rate_limit_skipped_total{dimension="token_budget_<dimension>"}. per_api_key needs an authenticated identity, per_user needs a resolved user id, per_team needs a resolved team id, per_client needs a peer address, and per_model needs a resolved model name. A skipped dimension charges nothing, so a deployment whose only token budget is per_api_key charges no tokens to traffic it cannot attribute, and the shared dimensions bound that traffic only when they are configured too. whitelist and bypass_keys exempt a request from the token budget along with every other limiter dimension; shared dimensions still bound the request. A scoped_bypass_keys entry waives only the periods it names, so a key scoped to request_rate keeps its token budgets and a key scoped to tokens_per_minute keeps its day budget.

tokens_per_minute accepts integers from 0 to 100000000, and tokens_per_day from 0 to 100000000000, where 0 is unlimited. The Hub-synced key path enforces its own per-key input_tpm and output_tpm windows and control_plane.policy.minute_limits sharing; the local dimension is independent, and a request on a hub-bound key can be bounded by both. Design rationale and the recorded decisions are in Token-Per-Minute and Token-Per-Day Budgets.

Deployment-Total Token Budgets¶

With storage: memory, every replica runs its own token buckets, so a deployment of N replicas admits up to N times the configured tokens_per_minute / tokens_per_day. The configured number only acts as a deployment total when the router is told how many replicas share it. Four modes cover that:

rate_limiting:
  storage: memory
  budget_division:
    replicas: 4
  limits:
    global:
      requests_per_second: 1000
      burst_capacity: 2000
      tokens_per_minute: 4000000
      tokens_per_day: 40000000

• Every admission-time token dimension is divided: global, per_client, per_api_key, per_user, per_team, and each per_model entry hold configured / replicas per period, with integer floor. per_backend is not a token dimension and is untouched. The reservation, refusal, and settlement paths are unchanged.
• A positive budget never becomes unlimited: when the fleet is larger than the budget, a period's divided capacity floors to zero. That is an authoritative zero allowance which refuses every positive reservation, not the 0 = unlimited configuration convention.
• A Hub share wins per period: share_tokens_per_minute and share_tokens_per_day replace the divided capacity for their period, and a period the statement leaves absent keeps the local hint, so one period can be Hub-driven while the other stays on the config hint. An absent member, or a statement whose two shares are both absent (the authoritative clear), returns every period to the local hint. A pushed 0 is a zero allowance, not a clear. The router enforces the pushed share directly and never recomputes one from the deployment totals and the enrolled router count.
• Redis ignores the config hint: a shared Redis bucket already carries the deployment total, so budget_division is inert while a Redis backend is in use, and config validate warns when both are configured. A Hub share still applies while present.
• Hot reload: a changed budget_division.replicas, or a Hub push that changes a share, rebuilds the budgets and resets the token bucket maps, so the new capacity applies from the next window. A request already in flight settles its hold on the parameters it reserved with.
• Validation: config validate warns when the in-memory backend carries a nonzero admission-time token budget and no budget_division, because the effective limit is then per replica. replicas must be at least 1, and 1 is a legal no-op.

Files API Per-Operation Limits¶

In addition to the router-wide rate_limiting section, the /v1/files endpoints support dedicated per-operation, per-caller rate limits under files.rate_limiting:

files:
  enabled: true
  rate_limiting:
    enabled: true
    uploads_per_minute: 10        # POST /v1/files (default: 10)
    downloads_per_minute: 100     # GET /v1/files/{file_id}/content (default: 100)
    operations_per_minute: 1000   # GET /v1/files, GET/HEAD/DELETE /v1/files/{file_id} (default: 1000)

When files.rate_limiting.enabled is true, the router enforces three independent token buckets per caller:

• Validation: All three limits (uploads_per_minute, downloads_per_minute, operations_per_minute) must be greater than 0 at startup and hot reload.
• Per-caller buckets: Each caller gets separate files_upload, files_download, and files_operations buckets keyed by the SHA-256 hash of the API key (Authorization: Bearer or x-api-key), the authenticated user ID, or the client IP.
• Headers and 429 responses: Every Files API response includes x-ratelimit-limit, x-ratelimit-remaining, x-ratelimit-reset, and their -requests variants. Exceeding a limit returns 429 Too Many Requests with Retry-After and error.details.limit_type set to the bucket category (files_upload, files_download, or files_operations).

Response Headers¶

Success Response¶

When a request is within rate limits and a caller-scoped dimension (per_api_key or per_client) evaluated it:

HTTP/1.1 200 OK
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1640995200

The numbers describe the caller's own bucket. With only shared dimensions configured (global, per_user, per_team, per_model, per_backend), an admitted response carries no X-RateLimit-* at all, because a shared bucket's level is aggregate load rather than the caller's quota. See Header attribution and refusal precedence.

A per_api_key token budget adds its own family, X-RateLimit-Tokens-Limit, X-RateLimit-Tokens-Remaining, and X-RateLimit-Tokens-Reset, holding the bucket capacity, the tokens left after this request's reservation, and the Unix time at which the bucket is whole again. A per_api_key day budget adds the X-RateLimit-Tokens-Day-* variants for its own bucket. The token family is separate from the request-count headers because the two buckets are independent: a caller can sit near its request limit with a nearly empty token budget, or the other way round. per_client, per_user, per_team, per_model, and global token budgets are withheld for the same reason their request-count levels are.

Rate Limited Response¶

When rate limit is exceeded:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Limit: 20
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1640995230
Content-Type: application/json

{
  "error": {
    "message": "Rate limit exceeded. Try again in 30 seconds.",
    "type": "rate_limit_exceeded",
    "param": null,
    "code": "rate_limit_exceeded",
    "details": {
      "limit_type": "api_key",
      "limit": 20,
      "remaining": 0,
      "reset_time": 1640995230
    }
  }
}

Key behaviors on 429 responses:

• Anthropic error format: On /anthropic/* routes and the root-level /v1/messages aliases the body uses the Anthropic error envelope, so Anthropic SDKs recognize the refusal as a rate limit. Status, headers, and error.details are the same as above:

  {"type": "error", "error": {"type": "rate_limit_error", "message": "Rate limit exceeded. Try again in 30 seconds.", "details": {"limit_type": "api_key", "limit": 20, "remaining": 0, "reset_time": 1640995230}}}
  

• Retry-After floor: Retry-After is always at least 1 (never 0), preventing clients from immediately retrying an endpoint that just rejected them.

• error.details.limit_type: Identifies the dimension that refused the request: global, client, api_key, user, team, backend, model:<name>, or token_budget. When multiple admission mounts apply, see Header attribution and refusal precedence for which dimension is reported.
• Token-budget refusals (limit_type: "token_budget"): An exhausted token bucket names the budget to raise in error.details.token_dimension (global, client, api_key, user, team, or model). A per_api_key refusal also reports that bucket's limit and remaining, in the body and in x-ratelimit-tokens-*; a global, client, user, team, or model refusal withholds them, because more than one credential draws on that bucket. Retry-After is size-aware: it reports the time the exhausted budget needs to refill the refused request's own cost. When the token check itself runs on Redis and Redis is unreachable, rate_limiting.redis.fallback_to_memory: false answers 503 with Retry-After: 5 instead, like the request-count dimensions.
• Per-backend refusals (limit_type: "backend"): Emitted by the routing layer when all candidate backends are at capacity (see Per-Backend Limits). The response includes the standard JSON error body and Retry-After, but omits X-RateLimit-* headers because the exhausted bucket belongs to the backend fleet rather than the caller.
• Browser and CORS handling: Identity-dimension 429 responses are emitted inside the CORS layer and include Access-Control-Allow-Origin. Because the global mount sits outside the CORS layer, a 429 without Access-Control-Allow-Origin is reported by browsers as an opaque network error rather than a readable 429 status.

Monitoring¶

Metrics¶

Rate limiting activity is tracked in Prometheus metrics:

# Total rate limit checks performed
rate_limit_checks_total 10342

# Requests rejected, labeled by limit dimension
rate_limit_exceeded_total{limit_type="per_client"} 42

# Current token bucket levels and capacity
rate_limit_tokens_remaining{bucket_type="per_client",identifier="..."} 15
rate_limit_bucket_capacity{bucket_type="per_client"} 20

# Whitelisted and bypassed (API key) requests
rate_limit_whitelisted_requests_total 123
rate_limit_bypassed_requests_total 45

# Configured dimensions skipped because the request carried no identity for
# them (e.g. per-client on a Unix-socket listener, per-key for an anonymous
# caller). A steadily growing counter means a configured limit is not binding.
rate_limit_skipped_total{dimension="client"} 7

# Token-budget activity (tokens_per_minute). outcome is reserved, refunded,
# or charged; refusals are additionally counted on
# rate_limit_exceeded_total{limit_type="token_budget"}.
rate_limit_token_budget_tokens_total{dimension="api_key",outcome="reserved"} 120000
rate_limit_token_budget_refusals_total{dimension="global"} 2

# Concurrent in-flight caps: holds per tenant (the series is removed at zero),
# and refusals, which also count on
# rate_limit_exceeded_total{limit_type="inflight"}.
rate_limit_inflight_requests{dimension="user",tenant="..."} 3
rate_limit_inflight_refusals_total{dimension="user",tenant="..."} 1

# Redis-backed checks that found the shared store unreachable (storage: redis).
# action="fallback": enforced on this replica's bucket (fail-open, the default);
# action="refused": answered 503 (redis.fallback_to_memory: false).
rate_limit_redis_unavailable_total{dimension="global",action="fallback"} 3

Metric aggregation across replicas¶

Every rate-limit family and the data-plane budget families (router_budget_*, see Budgets) falls into one of four aggregation classes. A per-replica enforcement counter is incremented where this replica makes the decision, so the fleet value is the sum, and rate()/increase() windows absorb restarts and scale events. A process-local gauge describes this process's own memory, so a sum is meaningless. A histogram is aggregated by summing its buckets across replicas, then taking the quantile.

When the bucket itself is shared (storage: redis), rate_limit_tokens_remaining is a shared-state view: every replica observes the same bucket, so summing the series double-counts, and one series or max by (bucket_type) reports the deployment's level. The in-memory backend keeps a bucket per replica, so each series describes only that replica's share.

Alert on an aggregate, not on a single replica's series. A fixed threshold on one series fires spuriously as replicas are added or removed: the same fleet traffic is split across more series, or one series disappears, while the fleet total is unchanged. Sum the counter series before rate(), and sum the histogram buckets before histogram_quantile; never average per-replica quantiles.

# Counter: aggregate the fleet, never one replica's series.
sum by (limit_type) (rate(rate_limit_exceeded_total[5m])) > 1

# Histogram: sum the buckets across replicas, then take the quantile.
histogram_quantile(
  0.95,
  sum by (le, tenant_dimension) (rate(rate_limit_fair_queue_wait_seconds_bucket[5m]))
) > 5

Logging¶

Rate limit events are logged with context:

{
  "level": "warn",
  "msg": "Rate limit exceeded",
  "client_id": "abc123...",
  "endpoint": "/v1/chat/completions",
  "limit_type": "burst",
  "limit_value": 20,
  "window": "5s"
}

Bypass Mechanisms¶

IP Whitelist¶

Whitelist trusted IP addresses or CIDR ranges:

rate_limiting:
  whitelist:
    - "192.168.1.0/24"      # Internal network
    - "10.0.0.1"            # Admin server
    - "172.16.0.0/16"       # Corporate network

API Key Bypass¶

Certain API keys can bypass all rate limits:

rate_limiting:
  bypass_keys:
    - "admin-key-123"
    - "monitoring-key-456"
    - "load-test-key-789"

Health Check and Metrics Exemption¶

The liveness and readiness paths are always exempt, so a saturated router keeps answering its own probes:

• /health
• /health/ready
• /healthz

The metrics endpoint is exempt as well, at whatever metrics.path is configured and only while metrics.enabled is true. With metrics disabled, /metrics is metered like any other unmatched path. Because the exemption means the limiter will not bound a scrape flood, and because a scrape's cost grows with metric cardinality, set metrics.auth and restrict the endpoint at the network level.

Bucket Capacity¶

Each per-identity bucket map (per_client, per_api_key, per_user, per_team, per_backend) holds up to 100,000 entries, with inactive entries expiring after 600 seconds. If a map remains full after expired entries are pruned, that dimension is skipped for the request: the request is admitted and all other configured dimensions continue to enforce.

Failing open on table saturation prevents denial-of-service via source-address churn (for example, rotating through 100,000 addresses in a single IPv6 /64 subnet to lock out new callers for 600 seconds). Every skipped evaluation increments rate_limit_skipped_total{dimension} and emits a warning log rate-limited to once per minute.

Storage Backends¶

Memory Storage (Default)¶

In-memory storage is fast but not shared across instances:

rate_limiting:
  storage: memory

Pros:

• No external dependencies
• Low latency
• Simple setup

Cons:

• Not shared across router instances
• Lost on restart
• Limited to single-instance deployments

Redis Storage (Distributed)¶

Redis storage enables distributed rate limiting across multiple router instances:

rate_limiting:
  storage: redis
  redis:
    url: "redis://localhost:6379"
    key_prefix: "continuum:ratelimit:"   # Prefix for rate limit keys (default)
    ttl: 3600                            # TTL for rate limit keys in seconds

Credentials can be embedded in the URL (redis://:password@host:6379). The rate_limiting.redis.url field does not expand ${VAR} references; render a complete URL before validation.

Pros:

• Shared across all router instances
• Persistent across restarts
• Accurate global limits

Cons:

• Requires Redis infrastructure
• Slightly higher latency
• Additional operational complexity

Store outages¶

Every limiter that uses the Redis store (global, per-API-key, per-client, per-model, and per-backend) follows the same outage policy:

• Fail-open (default): If Redis becomes unreachable, each replica enforces the check on its local in-memory bucket instead of disabling enforcement, so a fleet of N replicas admits up to N times the configured rate until Redis recovers. Counted on rate_limit_redis_unavailable_total{action="fallback"}.
• Fail-closed (redis.fallback_to_memory: false): The check is refused with 503 service_unavailable and Retry-After: 5. Counted on rate_limit_redis_unavailable_total{action="refused"}, not as a quota violation.
• Bounded checks: A Redis check that cannot reach the store is bounded (250 ms pool timeouts, 500 ms per check). After such a failure the router skips Redis for 5 seconds and applies the configured policy at once, then lets a single request probe Redis again; a successful probe resumes distributed checks.

Hot Reload Support¶

Rate limiting configuration supports hot reload for immediate updates:

# These settings update immediately without restart
rate_limiting:
  enabled: true                  # ✅ Immediate: Enable/disable rate limiting
  limits:
    per_client:
      requests_per_second: 10    # ✅ Immediate: New limits apply immediately
      burst_capacity: 20         # ✅ Immediate: Burst settings update instantly
      max_in_flight_requests: 8  # ✅ Immediate: The new cap applies to the next admission
    per_backend:
      requests_per_second: 100   # ✅ Immediate: Backend limits update instantly

Best Practices¶

1. Start Conservative¶

Begin with stricter limits and relax them based on actual usage:

rate_limiting:
  limits:
    per_client:
      requests_per_second: 5      # Start low
      burst_capacity: 10

2. Monitor and Adjust¶

Use metrics to understand actual traffic patterns:

• Track the rate_limit_exceeded_total metric
• Identify legitimate vs. abusive traffic
• Adjust limits based on data

3. Use Different Tiers¶

Implement tiered rate limits for different user classes with per-key overrides (requests per minute):

api_keys:
  api_keys:
    - key: "free-tier-key"
      rate_limit: 60

    - key: "pro-tier-key"
      rate_limit: 600

    - key: "enterprise-tier-key"
      rate_limit: 6000

4. Protect Expensive Models¶

Apply stricter limits to costly models with per_model limits:

rate_limiting:
  limits:
    per_model:
      gpt-5.6-sol:
        requests_per_second: 2
        burst_capacity: 5

Name the model the way your clients name it. The bucket keys on the client-facing name in the request path or JSON body, so an entry for an alias target or for an experiment variant does not bound traffic that arrives under a different client-facing name. See Which name the bucket keys on.

5. Use Redis for Production¶

For multi-instance deployments, use Redis:

rate_limiting:
  storage: redis
  redis:
    url: "redis://redis-cluster:6379"

Troubleshooting¶

Common Issues¶

Rate limits not applying¶

Symptom: Clients can exceed configured limits

Solutions:

1. Check if client is whitelisted
2. Verify rate_limiting.enabled is true
3. Check logs for rate limiting initialization
4. Ensure API key format is correct

Too many false positives¶

Symptom: Legitimate traffic being rate limited

Solutions:

1. Increase burst_capacity for bursty traffic
2. Review client identification (may be grouping multiple clients)
3. Consider using per-API-key limits
4. Add legitimate IPs to whitelist

Redis connection issues¶

Symptom: Rate limiting not working with Redis storage

Solutions:

1. Verify Redis connectivity
2. Check Redis authentication
3. Review connection pool settings
4. Monitor Redis performance

Debug Logging¶

Enable debug logging to see rate limiting decisions in the logs:

logging:
  level: debug

Related Documentation¶

• Rate Limiting Architecture - Implementation details and design decisions
• Configuration Guide - Complete configuration reference
• Admin API - Runtime configuration management
• Metrics - Monitoring and observability
• Error Handling - Error codes and retry strategies