Fair Queuing Across Tenants¶

Fair queuing is shipped (issue #2397, epic #2362). It is opt-in and off by default: with rate_limiting.fair_queuing absent or enabled: false, an exhausted dimension still answers 429 immediately, exactly as before.

What it changes¶

Without the mode, a refusal is first-come-first-served: the global dimension rejects when its bucket is empty, per_backend is a selection-time filter (an empty backend is dropped from the candidate set, 429 only when every candidate is exhausted), and nothing in the rate-limit middleware holds or schedules requests (src/http/middleware/rate_limit_v2/store.rs keeps buckets only). Under saturation one heavy tenant's flood consumes the freed capacity before anyone else can retry.

With the mode enabled, a request an identity-bearing admission dimension (per_api_key, per_user, per_team, per_model) refuses is held in its tenant's bounded queue instead of being answered immediately, and the queues are served by weighted fair scheduling. When its turn comes the request re-checks the limiter; if the check passes it is admitted, and if the dimension still refuses it re-queues within its budget.

The queue never dispatches a request it cannot admit within the budget, and queued time counts against the request's own total timeout: the effective admission budget is the smaller of max_wait_ms and the smallest configured request total timeout (timeouts.request.*.total), so a queued request is refused rather than served after its timeout.

Which refusals are queued¶

Queued: 429 refusals from the identity dimensions on the authenticated mount, that is limit_type api_key, user, team, or model.
Immediate, unchanged: the global and per-client refusals, because their mounts run before authentication and no tenant identity exists there yet; a 503 from a fail-closed store outage (not the caller's quota); and every refusal on a router with the mode disabled.
Not affected: the per-backend dimension, which is a selection-time candidate filter charged at dispatch, after admission.

A request that carries none of the user, team, or key identities is not queued, because it cannot be attributed to a tenant; it keeps the immediate 429.

Tenant identity and weights¶

tenant_dimension names the identity the queue and its weight are keyed on: user (the default), api_key, or team. The identity is resolved from the same resolution the limiter already uses (issue #2381): the hashed user id, the hashed team id (the Hub placeholder a team-less key carries is not an identity), or the key's stable secret-free identity. When the configured identity is absent, the resolver falls back in the fixed order user, then team, then api_key, so the same request always lands in the same queue.

weights gives a tenant a relative share of the admission attempts; default_weight (default 1) applies to every tenant without an entry. A tenant at weight 3 is served three times as often as one at weight 1. Each entry names the raw identity (a user id, a team id, or an API key; ${ENV} interpolation applies) and is used only to match the weight: the queue key itself is the hash, never the raw value.

Scheduling is smooth weighted round-robin over the tenants whose next waiter is ready (src/http/middleware/rate_limit_v2/fair_queue.rs). The scheduler issues at most max_in_flight attempts at once (default 8), so fairness does not serialize the whole admission path.

Bounds¶

Every level is bounded, so overload cannot turn admission into an unbounded buffer:

Key	Default	Bound
`max_queue_depth_per_tenant`	32	Requests queued for one tenant
`max_total_queued`	4096	Requests queued across the process
`max_tenants`	1024	Distinct tenant queues tracked at once
`max_in_flight`	8	Admission attempts the scheduler lets run at once
`max_wait_ms`	5000	Longest time one request may spend queued

A request that does not fit a bound is refused immediately with the position it would have had, and a request whose Retry-After cannot fit the remaining budget is refused instead of queued.

Position and wait estimate¶

A queued request exposes its queue position and how long it waited. A request refused after waiting (or refused at enqueue) carries x-fair-queue-position and x-fair-queue-wait-ms, and its error.details.fair_queue names the tenant dimension, the reason (queue_full, tenant_queue_full, tenant_table_full, wait_exceeds_budget, deadline_exceeded), the position, and the waited time. An admitted request that waited carries x-fair-queue-position, x-fair-queue-wait-ms, and x-fair-queue-estimated-wait-ms on its response. The estimate is computed at enqueue from the limiter's Retry-After plus the measured grant spacing behind the request.

Metrics¶

rate_limit_fair_queue_depth: requests currently held in the queues, sampled at every queue transition.
rate_limit_fair_queue_tenants: tenant queues the scheduler currently tracks.
rate_limit_fair_queue_wait_seconds{tenant_dimension}: time a request spent queued before its attempt was granted.
rate_limit_fair_queue_grants_total{tenant_dimension}: admission attempts granted from a queue.
rate_limit_fair_queue_refused_total{tenant_dimension, reason}: requests refused instead of, or after being, queued.

Composition with deferred admission¶

The two modes cover disjoint refusal classes, so a configuration may enable both:

Fair queuing handles request-count refusals at admission (api_key, user, team, model).
rate_limiting.deferred_admission handles token-budget refusals, inside the token reservation that runs only after the request-count check passes.

A request can therefore be fair-queued first and then, once admitted on the request-count dimensions, deferred on its token budget. The waits do not stack beyond the request's total timeout: fair queuing caps its wait at the smaller of max_wait_ms and the request timeout, and a token-budget refusal that cannot fit the remaining budget is answered with its size-aware Retry-After. A config that enables both is valid; neither mode changes the other's default-off behavior.

Multi-replica scope¶

The queue is per router process, like the deferred-admission hold count. Each replica schedules its own tenants, so fairness is exact within a replica and approximate across a fleet: a tenant's share across N replicas depends on how its requests distribute across them. A shared Redis queue was considered and left out of scope; the open questions that remain are recorded below.

Open questions¶

Whether the queue should live in the limiter's shared Redis store for multi-replica fairness, and how a queue position would be reported across replicas.
Whether the per-tenant weight should also come from the Hub tier instead of only the local weights list (tier priority is issue #2406).