Skip to content

Size-Aware Retry-After and Deferred Admission: Design Record

This note records the design of the size-aware Retry-After on token-budget refusals and the optional deferred admission mode (issues #2423, #2401). Both ship in the local rate_limiting limiter; this page records why they are built the way they are.

Context

Before the token-budget dimension, the limiter counted requests: a refusal's Retry-After was the deficit against a fixed per-request cost divided by the refill rate (token_bucket.rs), so a 200k-token request and a 10-token request received the same wait. The only feedback was an immediate 429.

Related behavior that ships alongside:

  • Hub-synced keys: the control-plane path enforces per-key input_tpm and output_tpm windows, and control_plane.policy.minute_limits.reserve_tokens reserves each request's estimated tokens at admission (see Hub tier minute limits). The local rate_limiting token dimension this design covers is independent of those windows.
  • Retry signaling on the Anthropic routes: a router-produced 408, 409, 429, or 5xx carries x-should-retry: true beside retry-after (#2477), so the size-aware wait reaches Claude clients through the header they already read.

Design

  • Token-budget dimension (#2401, shipped): the limiter carries a token-budget window beside the request-count one, fed by the same admission estimate the fallback budget planner uses (the typical whole-request input-token estimate, max_tokens included). The bucket's try_consume reports the wait for the estimated cost, and the per-request rate-limit headers carry both dimensions.
  • Size-aware Retry-After (#2423, shipped): Retry-After is computed from the refused request's own estimated cost against the exhausted budget, not from the fixed request cost. A client that honors it gets a backoff proportional to what it will consume. A request larger than the whole burst also receives a size-aware hint, and it is never deferred, because waiting cannot make it fit. The hint is floored at 1 second and capped at 86400.
  • Deferred admission (#2423, shipped, off by default): a refused request the limiter could admit within a short wait is held for that wait instead of being rejected, bounded by a configurable maximum so held requests cannot pile up open connections in the router. A held request has not been dispatched, so it holds no backend slot, and it holds no token reservation while waiting: the reservation is taken only once a wait succeeds. Requests the budget cannot admit within the bound still get an immediate 429 with the size-aware wait.
  • Reconciliation at completion (#2419, shipped): when a response finishes, the actual usage replaces the admission estimate in the budget, so the next estimate rides on what the previous requests really consumed.

Resolved decisions

  • The deferred-admission wait is bounded twice: per request by rate_limiting.deferred_admission.max_wait_ms (default 2000), measured from the first deferral, and per router process by max_held_requests (default 1000), a count on the rate-limit store that each replica keeps for itself and a client cancellation returns through the hold guard's Drop.
  • The two resets are separate rows: the request-count x-ratelimit-reset stays on its own window, and the token budget reports X-RateLimit-Tokens-Reset in its header family.