Skip to content

Rate Limiting

Continuum Router rate-limits requests to prevent abuse, allocate resources fairly, and protect backends from overload. The rate limiting system uses a token bucket algorithm with multiple layers of protection.

Overview

The router implements multi-tier rate limiting:

  • Per-client limits: Prevent individual clients from overwhelming the system
  • Per-backend limits: Protect individual backend services from overload
  • Per-API-key limits: Apply limits per authenticated key, with per-key overrides
  • Per-model limits: Protect specific (expensive) models
  • Global limits: Ensure overall system stability

Where limits apply

Every dimension is mounted at the outermost point where the identity it keys on already exists, and each is enforced at exactly one mount:

  • Global (limits.global): a single application-wide mount, outside authentication and CORS. Every request is charged exactly once: API requests (including those authentication later rejects with 401, so API-key guessing is metered), admin requests (including failed admin authentication), /status, /version, WebUI, Files, metrics, CORS preflights, and the 404 fallback.
  • Per-client (limits.per_client): mounted on the API routes (/v1/*, /anthropic/*, /embed_sparse) outside authentication. The dimension keys on the peer address, which the connection supplies before authentication runs, so it bounds an unauthenticated flood from a single source rather than leaving that traffic to the shared global bucket.
  • Identity dimensions (per_api_key, per_model, per_backend): mounted on the API routes inside authentication, where the AuthContext those dimensions need exists. This mount never touches the global or per-client buckets.

The request path on an API route therefore reads global -> per-client -> authentication -> per-key -> handler. Because the mounts enforce disjoint dimensions, no request is ever counted twice against any bucket, whatever route it takes.

The global mount exempts the two liveness paths, /health and /healthz, unconditionally: a saturated router must still answer its own probe, or the orchestrator restarts it in the middle of the overload it is shedding. Both serve static bodies, so exempting them costs nothing.

The metrics endpoint is also exempt, but the exemption is derived rather than fixed: it follows the effective metrics.path and applies only when metrics.enabled is true. A router with metrics disabled therefore meters /metrics like any other unmatched path, and an endpoint moved to a custom metrics.path keeps its exemption instead of being shed during exactly the incident you need it for.

Keeping scrapes un-sheddable is a deliberate trade. A scrape of an enabled metrics endpoint rebuilds the backend gauges and serializes the whole registry, so its cost grows with metric cardinality, and the limiter will not bound a flood of them. Protect the metrics endpoint with metrics.auth and network policy. Unless metrics.auth is configured the endpoint is unauthenticated, which combined with the exemption makes it the cheapest way to load a router from outside.

One consequence worth noting: unauthenticated and unrouted traffic (401-rejected requests, garbage paths hitting the 404 fallback) draws on the same limits.global bucket as API traffic, so a flood of it is bounded but can also shed API requests that share the bucket. Give limits.global headroom above your expected API rate, or front the router with a proxy that drops such traffic earlier. per_client narrows this for API paths specifically, since a flood from one address is shed on that address's own bucket before it can drain the shared one, but traffic to non-API paths is still bounded by limits.global alone.

Header attribution and refusal precedence

Several mounts can have an opinion about one request, so these rules pin down what the client sees:

  • On success, the x-ratelimit-* headers report only quota tied to an authenticated identity, which today means per_api_key. The caller has proved they hold the key the bucket is keyed on, so the numbers tell them nothing they do not already own. When the per-key dimension did not evaluate the request, the response carries no x-ratelimit-* at all.
  • Shared buckets are never reported on an admitted response. limits.global, per_model, and per_backend are drawn on by the whole fleet, and per_client is drawn on by everyone behind that source address (a NAT, an egress gateway, a corporate proxy), so none of their levels is the caller's own quota. per_client is additionally evaluated before authentication, where the router cannot name whom it would be reporting to. All four still enforce exactly as configured; they are simply not advertised. If you want your clients to see quota headers, configure per_api_key.
  • On refusal, the refusing dimension's numbers are authoritative: error.details.limit_type, retry-after, and x-ratelimit-* all describe the dimension that shed the request, including a shared one, because a client needs the retry hint and "the bucket is empty" is already implied by the 429 itself. No other mount restates its own numbers over them. When the global dimension and another would both refuse, the client sees the global refusal, because the global mount runs first on the request path. Retry-After is at least 1 on every path.
  • A refusal is readable in a browser whichever mount produced it. When CORS is enabled, retry-after and the x-ratelimit-* headers are always added to Access-Control-Expose-Headers, so cross-origin script can read the retry hint rather than only the status; and the global mount, which sits outside the CORS layer so preflights stay chargeable, echoes the allow-origin header itself on its short-circuited 429 instead of leaving it an opaque network error.

A request refused by a per-key limit has still consumed one token of global budget and one of its address's per-client budget: both were charged on the way in, before the inner mount refused it.

What the headers do and do not reveal

x-ratelimit-* describes a bucket keyed on an identity the caller authenticated as, so reading it discloses nothing they do not already own. Every response the router produced without evaluating that dimension carries no quota headers: a 401, a 404 from the fallback, an unauthenticated 200 such as /version, any response to a whitelisted address or a bypass_keys holder, and any response to an unauthenticated caller on an API path. That is uniform, so none of them is a signal.

The uniformity matters most where it is easiest to lose. The per-client mount runs before authentication, so it evaluates requests that authentication then rejects. If its quota were reported, a bypass_keys holder would get the exemption sentinel and no headers while any other token got real ones, on two otherwise identical 401s: header presence alone would confirm a suspected bypass key at one request per guess. That is why per-client quota is withheld rather than reported, and a test pins it.

One residual is accepted and worth stating plainly. In permissive auth mode with per_api_key configured, a response to a recognized key carries per-key headers while a response to an unrecognized token carries none, so the presence of the headers still distinguishes the two. Suppressing them for unrecognized callers does not help: absence would then be the signal instead, which is the same bit of information.

The decision is to accept it, because permissive mode is a development and trusted-network convenience, not an authentication boundary. In that mode the router already serves a valid and an invalid key identically: same models, same backends, same access. An attacker enumerating keys is after access, and access is already granted, so learning that a key is recognized has little marginal value there. If key validity is sensitive in your deployment, set api_keys.mode: blocking. An unrecognized key is then rejected with 401 before any identity dimension runs, and the 401 is the only thing the response says.

Table of Contents

Configuration

Basic Configuration

rate_limiting:
  enabled: true
  storage: memory  # or "redis" for distributed setups

  limits:
    per_client:
      requests_per_second: 10
      burst_capacity: 20
    per_backend:
      requests_per_second: 100
      burst_capacity: 200
    global:
      requests_per_second: 1000
      burst_capacity: 2000

Per-API-Key Rate Limiting

Two mechanisms apply to API keys:

  1. A per_api_key dimension inside rate_limiting.limits that acts as the default for every recognized key that has no per-key override of its own:

    rate_limiting:
  limits:
    per_api_key:
      requests_per_second: 10
      burst_capacity: 20

  2. A per-key rate_limit override (requests per minute) on individual entries in the api_keys section. When set, it replaces the per_api_key default for that key; rate_limit: 0 disables per-key limiting for that key entirely:

    api_keys:
  api_keys:
    - key: "premium-user-key"
      name: "Premium User"
      rate_limit: 600        # requests per minute

    - key: "standard-user-key"
      name: "Standard User"
      rate_limit: 60         # requests per minute

Per-key limits bind only to recognized keys: the dimension reads the identity that authentication attaches. Anonymous callers (no key, or an unknown key) are never per-key limited, in either auth mode. They are still bounded by per_client and global, which are mounted outside authentication and so apply whether or not the request is later rejected with 401.

Bypass Configuration

Certain clients can bypass rate limiting entirely:

rate_limiting:
  # Whitelist IPs that bypass rate limiting
  whitelist:
        - "192.168.1.0/24"
        - "10.0.0.1"

  # API keys that bypass rate limiting
  bypass_keys:
        - "admin-key-123"
        - "monitoring-key-456"

Client Identification

The router identifies clients using the following priority order:

  1. API Key from the Authorization: Bearer <token> or x-api-key header (preferred)

  2. Provides accurate tracking across different IPs

  3. X-Forwarded-For header (proxy/load balancer scenarios)

  4. Honored only when the request comes from a trusted proxy

  5. With use_rightmost_xff: true (default), the rightmost IP in the chain is used, which is harder to spoof

  6. X-Real-IP header (alternative proxy header)

  7. Also honored only from trusted proxies

  8. Direct IP address (when no proxy headers present)

  9. Used when the request comes directly to the router

Trusted Proxy Configuration

rate_limiting:
  # Proxies allowed to set X-Forwarded-For / X-Real-IP headers
  trusted_proxies:
    - "10.0.0.0/8"
    - "192.168.1.1"
  # Use the rightmost IP in the X-Forwarded-For chain (default: true)
  use_rightmost_xff: true

Forwarding headers from untrusted sources are ignored, so clients cannot evade per-client limits by spoofing X-Forwarded-For.

Unix Socket Listeners

A Unix domain socket peer has no IP address, so on unix: listeners the per-client dimension and the IP whitelist cannot bind and are skipped gracefully. Each skip is recorded in the rate_limit_skipped_total{dimension="client"} metric and a debug log. Global and per-API-key limits enforce normally on such listeners.

Rate Limiting Strategies

Token Bucket Algorithm

The router uses the token bucket algorithm, which allows for burst traffic while maintaining long-term rate limits:

  • Bucket capacity: Maximum number of tokens (burst_capacity)
  • Refill rate: Tokens added per second (requests_per_second)
  • Token cost: Each request consumes one token

How It Works

  1. Each client starts with a full bucket of tokens
  2. Tokens are consumed with each request
  3. Tokens refill at a constant rate
  4. Requests are rejected when bucket is empty

Per-Model Limits

In addition to the per-client, per-backend, per-API-key, and global dimensions, limits can target individual models. This is useful for protecting expensive models while leaving cheaper ones unconstrained:

rate_limiting:
  limits:
    per_model:
      gpt-4:
        requests_per_second: 2
        burst_capacity: 5
      gpt-3.5-turbo:
        requests_per_second: 20
        burst_capacity: 50

Response Headers

Success Response

When a request is within rate limits and a caller-scoped dimension (per_api_key or per_client) evaluated it:

HTTP/1.1 200 OK
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1640995200

The numbers describe the caller's own bucket. With only shared dimensions configured (global, per_model, per_backend) an admitted response carries no X-RateLimit-* at all, because a shared bucket's level is aggregate fleet load rather than the caller's quota. See Header attribution and refusal precedence.

Rate Limited Response

When rate limit is exceeded:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Limit: 20
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1640995230
Content-Type: application/json

{
  "error": {
    "message": "Rate limit exceeded. Try again in 30 seconds.",
    "type": "rate_limit_exceeded",
    "code": 429,
    "details": {
      "limit_type": "api_key",
      "limit": 20,
      "remaining": 0,
      "reset_time": 1640995230
    }
  }
}

Retry-After is always at least 1: a refused request never advertises Retry-After: 0, so clients cannot be told to retry immediately against an endpoint that has just refused them. error.details.limit_type names the dimension that shed the request (global, client, api_key, backend, or model:<name>). Both mounts emit the same throttling headers and body shape; see Header attribution and refusal precedence for which dimension's numbers appear when more than one applies. One difference matters for browser clients: the global mount sits outside the CORS layer, so its 429 carries no Access-Control-Allow-Origin and a browser reports it as an opaque network error instead of a readable 429 the page can back off on. An identity-dimension 429 is emitted inside the CORS layer and does carry the header.

Monitoring

Metrics

Rate limiting activity is tracked in Prometheus metrics:

# Total rate limit checks performed
rate_limit_checks_total 10342

# Requests rejected, labeled by limit dimension
rate_limit_exceeded_total{limit_type="per_client"} 42

# Current token bucket levels and capacity
rate_limit_tokens_remaining{bucket_type="per_client",identifier="..."} 15
rate_limit_bucket_capacity{bucket_type="per_client"} 20

# Whitelisted and bypassed (API key) requests
rate_limit_whitelisted_requests_total 123
rate_limit_bypassed_requests_total 45

# Configured dimensions skipped because the request carried no identity for
# them (e.g. per-client on a Unix-socket listener, per-key for an anonymous
# caller). A steadily growing counter means a configured limit is not binding.
rate_limit_skipped_total{dimension="client"} 7

Logging

Rate limit events are logged with context:

{
  "level": "warn",
  "msg": "Rate limit exceeded",
  "client_id": "abc123...",
  "endpoint": "/v1/chat/completions",
  "limit_type": "burst",
  "limit_value": 20,
  "window": "5s"
}

Bypass Mechanisms

IP Whitelist

Whitelist trusted IP addresses or CIDR ranges:

rate_limiting:
  whitelist:
        - "192.168.1.0/24"      # Internal network
        - "10.0.0.1"            # Admin server
        - "172.16.0.0/16"       # Corporate network

API Key Bypass

Certain API keys can bypass all rate limits:

rate_limiting:
  bypass_keys:
        - "admin-key-123"
        - "monitoring-key-456"
        - "load-test-key-789"

Health Check and Metrics Exemption

Two liveness paths are always exempt, so a saturated router keeps answering its own probe instead of being restarted mid-overload:

  • /health
  • /healthz

The metrics endpoint is exempt as well, at whatever metrics.path is configured and only while metrics.enabled is true. With metrics disabled, /metrics is metered like any other unmatched path. Because the exemption means the limiter will not bound a scrape flood, and because a scrape's cost grows with metric cardinality, set metrics.auth and restrict the endpoint at the network level.

Bucket Capacity

Each per-identity bucket map (per_client, per_api_key, per_backend) holds at most 100,000 entries, with inactive entries expiring after 600 seconds. When a map is full after expiry pruning, that dimension is skipped for the request rather than refusing it: the request is admitted and the remaining dimensions still enforce.

This is deliberate. Refusing would save no memory, because the refusal path inserts no bucket either, and 100,000 distinct source addresses are easy to source from a single IPv6 /64, so failing closed would let an attacker turn a bounded map into a ten-minute outage for every new caller. The skip is visible in rate_limit_skipped_total{dimension} and in a warning rate-limited to one line per minute.

Storage Backends

Memory Storage (Default)

In-memory storage is fast but not shared across instances:

rate_limiting:
  storage: memory

Pros: - No external dependencies - Low latency - Simple setup

Cons: - Not shared across router instances - Lost on restart - Limited to single-instance deployments

Redis Storage (Distributed)

Redis storage enables distributed rate limiting across multiple router instances:

rate_limiting:
  storage: redis
  redis:
    url: "redis://localhost:6379"
    key_prefix: "continuum:ratelimit:"   # Prefix for rate limit keys (default)
    ttl: 3600                            # TTL for rate limit keys in seconds

Credentials can be embedded in the URL (redis://:password@host:6379). The rate_limiting.redis.url field does not expand ${VAR} references; render a complete URL before validation.

Pros: - Shared across all router instances - Persistent across restarts - Accurate global limits

Cons: - Requires Redis infrastructure - Slightly higher latency - Additional operational complexity

Hot Reload Support

Rate limiting configuration supports hot reload for immediate updates:

# These settings update immediately without restart
rate_limiting:
  enabled: true                  # ✅ Immediate: Enable/disable rate limiting
  limits:
    per_client:
      requests_per_second: 10    # ✅ Immediate: New limits apply immediately
      burst_capacity: 20         # ✅ Immediate: Burst settings update instantly
    per_backend:
      requests_per_second: 100   # ✅ Immediate: Backend limits update instantly

Best Practices

1. Start Conservative

Begin with stricter limits and relax them based on actual usage:

rate_limiting:
  limits:
    per_client:
      requests_per_second: 5      # Start low
      burst_capacity: 10

2. Monitor and Adjust

Use metrics to understand actual traffic patterns:

  • Track the rate_limit_exceeded_total metric
  • Identify legitimate vs. abusive traffic
  • Adjust limits based on data

3. Use Different Tiers

Implement tiered rate limits for different user classes with per-key overrides (requests per minute):

api_keys:
  api_keys:
    - key: "free-tier-key"
      rate_limit: 60

    - key: "pro-tier-key"
      rate_limit: 600

    - key: "enterprise-tier-key"
      rate_limit: 6000

4. Protect Expensive Models

Apply stricter limits to costly models with per_model limits:

rate_limiting:
  limits:
    per_model:
      gpt-4:
        requests_per_second: 2
        burst_capacity: 5

5. Use Redis for Production

For multi-instance deployments, use Redis:

rate_limiting:
  storage: redis
  redis:
    url: "redis://redis-cluster:6379"

Troubleshooting

Common Issues

Rate limits not applying

Symptom: Clients can exceed configured limits

Solutions: 1. Check if client is whitelisted 2. Verify rate_limiting.enabled is true 3. Check logs for rate limiting initialization 4. Ensure API key format is correct

Too many false positives

Symptom: Legitimate traffic being rate limited

Solutions: 1. Increase burst_capacity for bursty traffic 2. Review client identification (may be grouping multiple clients) 3. Consider using per-API-key limits 4. Add legitimate IPs to whitelist

Redis connection issues

Symptom: Rate limiting not working with Redis storage

Solutions: 1. Verify Redis connectivity 2. Check Redis authentication 3. Review connection pool settings 4. Monitor Redis performance

Debug Logging

Enable debug logging to see rate limiting decisions in the logs:

logging:
  level: debug