Rate Limiting¶

The router implements rate limiting to protect against abuse and ensure fair resource allocation across clients.

Architecture¶

Rate limiting is implemented at the HTTP layer using a dual-window approach:

pub struct ModelEndpointRateLimiter {
    cache: Arc<DashMap<String, (Instant, u32, u32)>>, // (window_start, request_count, burst_count)
    window_duration: Duration,        // 60 seconds for sustained limit
    max_requests_per_window: u32,     // 100 requests per minute
    burst_limit: u32,                 // 20 requests maximum
    burst_window: Duration,           // 5 seconds for burst detection
}

Client Identification¶

The system uses a priority-based client identification strategy:

1. API Key (Preferred): Uses the first 16 characters of Bearer token
2. IP Address (Fallback): Extracted from headers in order:
3. X-Forwarded-For (for proxy/load balancer environments)
4. X-Real-IP (alternative header)
5. Falls back to "unknown" if no IP can be determined

Rate Limit Rules¶

Each client is independently rate-limited with:

• Sustained Limit: 100 requests per minute (sliding window)
• Burst Protection: Maximum 20 requests in any 5-second window
• Per-Client Isolation: Each API key or IP has separate quotas

Configuration¶

rate_limiting:
  enabled: true

  # Global defaults
  default:
    requests_per_minute: 100
    burst_limit: 20
    burst_window_seconds: 5

  # Per-endpoint overrides
  endpoints:
    /v1/chat/completions:
      requests_per_minute: 60
      burst_limit: 10
    /v1/models:
      requests_per_minute: 300
      burst_limit: 50

  # Per-client overrides (by API key prefix)
  clients:
    sk-premium-*:
      requests_per_minute: 500
      burst_limit: 100
    sk-free-*:
      requests_per_minute: 20
      burst_limit: 5

Cache Management¶

The rate limiter includes automatic cleanup mechanisms:

• Memory Efficiency: DashMap for lock-free concurrent access
• Automatic Cleanup: Removes expired entries when cache > 1000 entries
• TTL Differentiation:
  • Empty model responses: 5-second cache TTL (DoS prevention)
  • Normal responses: 60-second cache TTL

Security Considerations¶

The rate limiting system addresses several security concerns:

1. DoS Prevention: Short TTL for empty responses prevents cache poisoning
2. Fair Resource Allocation: Per-client limits prevent monopolization
3. Burst Protection: Dual-window approach catches both sustained and burst attacks
4. Client Spoofing Mitigation: API key prioritization over IP addresses

Metrics and Monitoring¶

Rate limiting integrates with the metrics system:

pub struct ModelMetrics {
    pub rate_limit_violations: AtomicU64,  // Track rejected requests
    pub empty_responses_returned: AtomicU64, // Monitor empty response rate
    pub transient_errors: AtomicU64,       // Network/timeout failures
    pub permanent_errors: AtomicU64,       // Auth/config failures
}

Prometheus Metrics¶

# Rate limit violations
rate_limit_violations_total{client_id="sk-abc...", endpoint="/v1/chat/completions"} 15

# Current request counts
rate_limit_current_requests{client_id="sk-abc...", window="minute"} 45
rate_limit_current_requests{client_id="sk-abc...", window="burst"} 8

# Rate limit status
rate_limit_remaining{client_id="sk-abc...", window="minute"} 55

Response Headers¶

Rate limit information is included in response headers:

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 55
X-RateLimit-Reset: 1699574460
X-RateLimit-Burst-Limit: 20
X-RateLimit-Burst-Remaining: 12

Error Response¶

When rate limited, the API returns:

{
  "error": {
    "message": "Rate limit exceeded. Please retry after 45 seconds.",
    "type": "rate_limit_exceeded",
    "param": null,
    "code": "rate_limit_exceeded",
    "details": {
      "limit": 100,
      "remaining": 0,
      "reset_at": "2024-01-15T10:30:00Z",
      "retry_after": 45
    }
  }
}

Implementation Details¶

The rate limiter is implemented as a singleton service:

static MODEL_RATE_LIMITER: once_cell::sync::Lazy<ModelEndpointRateLimiter> =
    once_cell::sync::Lazy::new(ModelEndpointRateLimiter::new);

This ensures: - Single point of rate limit enforcement - Consistent state across all request handlers - Minimal memory overhead

Multi-Tenant Limits Design¶

One tenant (a user, a team, or a leaked key) can currently monopolize shared backend capacity: the only limit dimensions are per-client, per-backend, global, per-API-key, and per-model, and the per-IP bucket is shared by everyone behind one NAT. The multi-tenant limits epic (#2362) bounds a single tenant's footprint with the pieces below, designed to compose with the existing rate_limiting config.

Note: Hub-synced API keys (the control-plane feature) already have per-key rpm, input_tpm, and output_tpm windows, enforced by src/control_plane/enforcement.rs. control_plane.policy.minute_limits adds opt-in shared Redis counters and admission-time token reservation for those windows (see Hub tier minute limits). The local rate_limiting limiter keeps its own token dimension (below), which applies to every key; it does not share the control-plane reservation code, and a request on a hub-bound key can be bounded by both.

Pieces¶

Rollout order¶

1. Per-user / per-team dimension: shipped, extending RateLimits with user and team blocks and keying the limiter on the API key's resolved identity (#2381, #2426).
2. Concurrent in-flight cap: shipped. A per-identity counter caps the streams a tenant holds at once and refuses over the cap with 429 (issue #2383).
3. Token budget: shipped, reserving at admission and reconciling at completion (#2395, #2419), including streamed completions at the stream's normal end (#2420).
4. Distributed visibility: alert on the shared-store degraded state with rate_limit_redis_unavailable_total{dimension, action}, which now exists (#2408).
5. Tier limit overrides: shipped, applying a per-tier rate, burst, and minute budget to the per-key dimension while the Hub's own minute windows stay separately enforced (#2406).
6. Fair queuing and tier priority: fair queuing is shipped (#2397), with per-tenant bounded queues served by weighted fair scheduling; tier priority remains.

Caller quota read¶

GET /v1/quota reports the same limits the response headers push, on request. It adds no limiter mount: the identity mount takes a read-only branch at CALLER_QUOTA_PATH, the Hub policy path skips admit_request, and both read through access that creates no entry (get rather than entry, and a cost-zero run of the Redis bucket script). services::key_budget::read supplies the long-window budget, and the handler merges the three readings.

A read charges no identity-bound bucket and creates no window, so a key at its per_api_key or max_tokens_budget limit can still learn its level. The shared global and per_client mounts still charge the path at the /v1/models cost of 0.1, but neither branch reads their levels. Rows carry scope: "fleet" only when a shared store answered, and scope: "replica" with degraded: true when a configured shared store was unreachable.

The document is restricted to the caller's identity: shared fleet levels, batch dimensions, and key or tenant identifiers are excluded. In permissive mode an unrecognized caller receives an empty document, so the endpoint adds no key-validity signal beyond the existing header behavior.

Concurrent in-flight cap¶

max_in_flight_requests (issue #2383) is a per-tenant ceiling on requests resident at once, on the admission-time tenant dimensions per_api_key, per_user, per_team, and per_client. 0, the default, is unlimited. Admission takes one slot per configured dimension after every request-count and token-budget check has passed, so a refused request takes none, and a request over the ceiling is refused with 429, limit_type: "inflight", error.details.inflight_dimension, and Retry-After: 1. It is never queued.

The slot is an RAII guard attached to the response body, the same lifecycle server.max_concurrent_requests uses: a stream holds its slot until its last frame, a handler failure releases it, and a dropped connection releases it with the request future. Accounting is per replica, so a fleet of N replicas enforces up to N times the configured cap, like the in-memory request buckets. The cap is ignored on global, per_backend, and per_model (config validate warns), and a full or scoped bypass can exempt the inflight class. Metrics: rate_limit_inflight_requests{dimension, tenant} (removed when the count reaches zero) and rate_limit_inflight_refusals_total{dimension, tenant}.

Token-Budget Limiting¶

The request-count dimensions charge a fixed per-path cost (/v1/chat/completions charges one token), so a minutes-long 200k-token generation holds the same rate-limit share as a 10-token request, and it frees that share the instant admission succeeds. tokens_per_minute (issues #2395 and #2419, epic #2362) meters tokens instead, on every admission-time dimension (per_api_key, per_user, per_team, per_client, per_model, global). tokens_per_day (issue #2327) adds a second, independent bucket per dimension beside the minute one, with the same reservation lifecycle: its capacity is one day of the rate, it refills at tokens_per_day / 86400 tokens per second as a leaky-bucket window (not a calendar-day reset), and a request must fit both budgets of every configured dimension.

Note: Hub-synced keys keep their own input_tpm and output_tpm windows and the shared counters of control_plane.policy.minute_limits (see Hub tier minute limits). The local dimension is independent of that path and applies to every key. The code lives in src/http/middleware/rate_limit_v2/token_budget.rs, token_reservation.rs, and store.rs.

Reservation lifecycle¶

• Admission: reserve input estimate + output limit against every configured dimension before dispatch, reusing the when.input_tokens estimator. All dimensions are reserved in one step at the identity layer, so the request is charged exactly once per dimension.
• Completion: replace the reservation with the usage the backend reported. Unused reservation is refunded and an overage is charged; a failure, a missing usage block, or a client cancellation releases the whole reservation when the last handle drops.
• Streaming: keep the reservation until the body ends, so a long generation does not free its share mid-flight. A stream that reaches its normal end settles against its finalized usage, and a truncated stream releases in full (#2420).
• Cache hits and retries: a response-cache hit releases the reservation because nothing is generated. A retry or fallback hop stays inside the request that took the reservation, so one reservation covers every attempt and settles once with the final reported usage.

A per-reservation expiry was rejected because it would also expire an honest generation running longer than a minute.

Storage¶

The token bucket is Redis-backed like the request-count bucket, so cross-replica deployments share one quota, and Redis unavailability follows rate_limiting.redis.fallback_to_memory, counted by rate_limit_redis_unavailable_total{dimension, action}. The minute bucket lives under the token_budget key namespace and the day bucket under token_budget_day (#2327), so the two never draw on one another; a day-bucket key raises its TTL to at least one day, because a key that expired sooner would hand back a fresh day's allowance. Each hold records the backend that served it, so a reservation taken on this replica's in-memory bucket during an outage is settled or released against that bucket, never against a Redis bucket that never held it.

Related Documentation¶

• Circuit Breaker - Standalone state machine and Admin controls
• Model Fallback - Fallback strategies
• Architecture Overview - Main architecture guide