Skip to content

Per-Tenant Concurrent In-Flight Cap: Design Record

This note records the per-tenant concurrent in-flight cap (issues #2383, epic #2362). It is implemented and off by default.

Background

Rate limiting is requests per second plus burst (RateLimitSettings): the bucket is consumed at request time and refills over the window, but it says nothing about how many of a tenant's requests are still streaming. Before this cap, the concurrency bounds were not per tenant:

  • server.max_concurrent_requests (src/http/middleware/admission.rs, #1173): one router-wide in-flight ceiling. A permit spans the whole request, including the whole stream, and a request over the ceiling is shed at once with 503 and Retry-After. One tenant can still take every permit.
  • fallback.fallback_policy.max_concurrent_dials_per_backend: dial-only, fallback hops only.
  • Per-backend HTTP/2 SETTINGS_MAX_CONCURRENT_STREAMS: one user can exhaust it; long streams fill the stream cap and further requests wait inside the HTTP client for minutes.

Per-key rate overrides exist (api_keys[].rate_limit, requests per minute), but they bound arrivals, not requests in flight. This page details step 2 of the multi-tenant limits rollout.

Shipped design

  • Config surface: max_in_flight_requests (integer, 0 = unlimited, the default) on the admission-time tenant dimensions per_api_key, per_user, per_team, and per_client. The field is ignored on global (use server.max_concurrent_requests for the router-wide residency bound), per_backend (no admission seam), and per_model (not a tenant identity); config validate warns when it is set on an ignored dimension.
  • In-flight tracking: a per-tenant counter map beside the bucket store, keyed by the same identity the request-count bucket for that dimension uses. Admission takes one slot per configured dimension after every request-count and token-budget check has passed, so a refused request takes no slot.
  • Release: the slots are held by an RAII guard that is attached to the response body, the same whole-request permit lifecycle server.max_concurrent_requests uses. A stream therefore holds its slot until the last frame; a handler failure or a client disconnect drops the guard and frees it. Accounting is per replica (see issue #2383's design notes): a distributed counter would need leases, and a crashed process could not release them, so a fleet of N replicas enforces up to N times the configured cap, like the in-memory request buckets.
  • Behavior beyond the cap: reject, never queue. A refusal is 429 with limit_type: "inflight", error.details.inflight_dimension naming api_key, user, team, or client, and a fixed Retry-After: 1 (the router does not estimate when a stream will end). Quota levels are reported only for per_api_key; the user, team, and client counts are shared and withhold them like every other shared bucket. Fair queuing remains a separate design (issues #2397 and #2423).
  • Bypass: a full whitelist / bypass_keys exemption skips the cap; a scoped bypass names the new inflight class (issue #2426).
  • Observability: rate_limit_inflight_requests{dimension, tenant} reports current holds (the series is removed at zero), rate_limit_inflight_refusals_total{dimension, tenant} reports refusals, and a refusal also counts on rate_limit_exceeded_total{limit_type="inflight"}.

Resolved questions

  • The default cap is 0 (off), and the team dimension keys on the local resolved team identity (organization_id), the same identity the per-team request-count bucket uses.
  • The refusal names the limit type in the same error.details envelope the per-backend cap uses.