Token-Per-Minute and Token-Per-Day Budgets¶

The tokens_per_minute and tokens_per_day dimensions limit how many tokens a request may consume, per minute and per day, on every admission-time rate-limit dimension (issues #2395, #2327, #2419). Both are shipped and off by default: 0 (the default) leaves a dimension unlimited on that period, and a config without the fields behaves exactly as before.

The request-count dimensions alone leave a gap. A few huge inputs from one tenant, or a reasoning model run at its maximum thinking budget, stay under any RPS cap while saturating the backends. The router already meters usage per request and computes a per-request input-token estimate (core::prompt_estimate, which routing_policies[].when.input_tokens matches against), and this dimension charges both against a bucket.

Hub-synced keys have a separate implementation of the same idea: the control-plane path enforces per-key input_tpm and output_tpm windows (src/control_plane/enforcement.rs), and control_plane.policy.minute_limits adds shared Redis counters with admission-time reservation (see Hub tier minute limits). The local dimension documented here is independent of that path; a request on a hub-bound key can be bounded by both.

Configuration surface¶

tokens_per_minute and tokens_per_day are fields of RateLimitSettings, beside requests_per_second, burst_capacity, and window_duration. Every admission-time dimension carries its own independent bucket per period:

per_backend is deliberately absent. That dimension is a selection-time candidate filter charged once per dispatch, after admission, so a token budget there has nowhere to attach; config validate warns when either field is set on it. per_model needs no such warning: it is an admission-time token dimension, charged against the resolved client-facing model name by the same reservation as the tenant dimensions, so its budget is enforced rather than ignored (issue #2899).

The minute budget's capacity is one minute of the configured rate, so a tenant may spend its whole minute at once. The continuous refill is tokens_per_minute / 60 tokens per second, which is why a duration-style window_duration plays no part.

The day budget (#2327) is a second, independent bucket beside the minute one, charged by the same admission reservation and completion reconciliation. A request must fit both budgets of every configured dimension. Its capacity is one day of the rate and the continuous refill is tokens_per_day / 86400 tokens per second, so a tenant may spend its whole day at once and the bucket is whole again one day after it was drawn down. It is a leaky-bucket window: it does not reset at midnight. This matches the request-count buckets' refill-bucket semantics extended to a day, and differs from the max_tokens_budget rolling windows of #2323, which are sliding-window quotas at the auth seam that reset usage continuously over 1d/7d/30d of history rather than refilling capacity.

Admission¶

A request on a text-generation route reserves its whole-request input estimate plus its output limit, once per configured dimension, before dispatch. The estimated and charged routes are:

• /v1/chat/completions
• /v1/completions
• /v1/responses
• /v1/responses/compact
• /v1/messages and /anthropic/v1/messages

The input half walks every message, tools, functions, and response_format, and adds the prompt fields the chat walk does not see (prompt, input, instructions, system) so the completions, Responses, and Anthropic shapes are charged for their input too. The output half is max_tokens, max_completion_tokens, or max_output_tokens, whichever is largest, and defaults to 4096 when the request states none. A float or numeric-string spelling (32000.0, "32000") reads as the same ceiling: pydantic-based upstreams coerce it and then generate up to it, so reading only JSON integers would under-reserve the request. Reserving the output limit rather than the generated length is what stops a request with no stated cap from holding only its (small) input.

The reservation is an over-estimate by construction: a client can pad the estimate and the output limit is a ceiling. Completion-time reconciliation turns the difference back into budget.

The reservation margin is zero beyond those two halves. The hold carries no extra percentage or fixed pad on top of the input estimate and the output ceiling, because the estimator is already generous (it walks the prompt without a tokenizer and adds a per-message floor) and reconciliation refunds whatever the request did not spend. A second margin would only hold budget the request never consumes, for the whole life of its reservation.

Between the store returning the reservation and the handle being inserted into the request extensions, the handle is a local owned by the admission future. A client cancellation in that window drops the future, which drops the handle and releases the hold, so the node that made the reservation is always the one that ends it; there is no window in which a reservation exists and nothing can reach it.

Rules at admission:

• Every configured dimension is charged, on both periods. The first dimension and budget that cannot cover the request refuses; the reservation already taken on earlier dimensions is released in full because the request never runs.
• A request larger than the whole burst is refused immediately. Waiting cannot make it fit, so admitting it would be a promise the bucket cannot keep. The refusal's Retry-After is still size-aware: it reports the time the budget needs to refill the request's own cost.
• Whitelist and bypass_keys exempt a request from the token budget too. The exemption covers every limiter dimension.
• A dimension with no identity for the request is skipped and counted on rate_limit_skipped_total{dimension="token_budget_<dimension>"}. per_api_key needs an AuthContext; per_user needs a resolved user id; per_team needs a resolved team id; per_client needs a peer address; per_model needs a resolved model. A skipped dimension charges nothing, so a deployment whose only token budget is per_api_key charges no tokens to traffic it cannot attribute; a shared dimension bounds that traffic only when it is configured too. Each valid key is its own identity with its own budget, so rotating keys moves load across budgets by design; per_user and per_team intentionally pool those keys.
• A body the peek cannot read is admitted unchanged. The route's own handler owns body validation and the error shape, and the peek replays every byte. The request makes no reservation, the same fail-open posture the dimension takes on a full bucket map.

The model extractor and the token reservation both need the request body. One bounded, replay-safe buffer serves both: the extractor stores it as a PeekedBody request extension while a token budget is configured, and admission removes and reuses it instead of reading the body again.

Reconciliation¶

The completion path replaces the reservation with the usage the backend reported (input plus output tokens):

• Less usage than reserved refunds the difference to every dimension.
• More usage than reserved charges the overage to every dimension. An overage the bucket cannot cover consumes nothing, rather than partially draining it; both backends behave the same way.
• No settlement (a failure, a response with no usage block, or a client cancellation) releases the whole reservation when the last handle drops.

A streamed completion settles when the stream reaches its normal end, against the finalized usage: the backend's usage block, or the input estimate plus the router's delta count when the backend reported none.

• A stream cut before its end (a client cancellation, a timeout, an upstream error, a guardrail block) never settles. Its reservation is released in full when the body guard drops.
• A mid-stream fallback hop never settles either. The successor attempt settles the reservation with the final reported usage.

Settlement and release never block the request task. The in-memory backend applies them on the current task and the Redis backend on a detached one. A settlement that cannot reach Redis returns the budget at the bucket's refill rate instead, which bounds the loss to one window.

Request deduplication splits ownership cleanly. Two identical in-flight requests each take their own admission reservation, because each is an independent admission; only the request that actually dispatched (the leader) settles its own reservation, against the usage the backend reported. A waiter made no backend call and replays the leader's usage, so its row settles nothing and its handle is released in full when its response body ends. No handle is shared between the two requests, so no reservation can be settled twice or released twice, and the leader's settlement is the request's only token charge.

Each hold records the backend that served it. A reservation taken on this replica's in-memory bucket while Redis was unreachable is settled or released against that bucket, never against a Redis bucket that never held it.

Request lifecycle¶

A per-reservation expiry was rejected because it would also expire the honest hold of a generation running longer than a minute, which the streaming rule requires keeping.

Refusal¶

An exhausted budget uses the existing throttling path: 429, limit_type: "token_budget", and error.details.token_dimension naming the dimension that refused (global, client, api_key, or model). When both periods are configured, the refusal also carries error.details.token_budget naming the exhausted budget (minute or day). A caller configured on several dimensions therefore learns which budget to raise.

A refusal by the identity-scoped api_key budget also carries that bucket's capacity and remaining tokens, in error.details and in x-ratelimit-tokens-*; a day-budget refusal carries them on x-ratelimit-tokens-day-*. A refusal by a shared dimension (global, client, model) names itself and carries Retry-After, but withholds the bucket's level from the body and from every x-ratelimit-* header: other tenants draw on that bucket, so a probe that fails must not report how full it is. The Retry-After is computed from the refused request's own estimate against the exhausted budget, so a caller that honors it backs off in proportion to what it will consume.

The request-count x-ratelimit-* headers still reach the response on a token refusal. They describe the caller's own request-count quota, an independent bucket that admitted this request before the token check ran, so the token refusal reports it unchanged and only withholds the refused token bucket's own level (issue #2899).

A Redis-backed token check that finds Redis unreachable follows rate_limiting.redis.fallback_to_memory: true (the default) admits and reserves on the replica-local bucket, and false (fail-closed) answers 503 with Retry-After: 5 until Redis answers. Either way the event is counted by rate_limit_redis_unavailable_total{dimension, action}.

When both periods are configured, their two reservations run as one Redis round trip instead of two: a single script checks both buckets and consumes from both only when both admit, so a minute refusal never drains the day budget (issue #2899). The script takes both keys in one call, which is safe because the Redis endpoint is a single server: Sentinel and Cluster URLs are rejected at config-load time.

Deployment-total budgets¶

Every replica carries its own token buckets, so a fleet of N replicas admits up to N times the configured allowance unless something tells one replica how much of the total it owns. Shared state is the general answer, and the Redis backend already provides it. Deployments that keep the in-memory backend need a division mechanism instead (issue #2911). This is the router side of the combined mechanism; the Hub computes and pushes per-router shares in the continuum-hub repository.

The divisor is resolved per period, most authoritative source first:

1. Hub share: deployment_token_budget.share_tokens_per_minute or share_tokens_per_day. A present share replaces the divided capacity for its period, and a period the statement leaves absent keeps the local hint, so the two periods are independent.
2. Config hint: rate_limiting.budget_division.replicas. Every admission-time token dimension's configured allowance is divided by this count, with integer floor.
3. One: today's per-replica budget, unchanged when neither source is present.

The three "no Hub share" wire forms behave identically: an older Hub omits the member, an authoritative clear carries a statement whose two shares are both absent, and a router with no applied envelope holds none. Each returns every period to the local hint. A pushed share of 0 is different. Integer division can legitimately assign zero tokens when the fleet is larger than the budget, so a zero share is an authoritative zero allowance: the router refuses traffic rather than falling back to the hint.

Zero allowance versus unlimited¶

The public configuration convention is 0 = unlimited, and a divided budget must not inherit it. The implementation therefore keeps a resolved allowance per period that distinguishes unlimited from a budget in force with zero capacity. Only a configured 0, no divisor source at all, or a bypass is unlimited; a positive configured value divided down to zero, or a pushed share of zero, refuses every positive reservation. The reservation and settlement paths are otherwise untouched: a zero-capacity period fails the existing over-burst check, which is the same refusal an oversized request already receives.

Scope of the division¶

The division covers the six admission-time token dimensions: global, per_client, per_api_key, per_user, per_team, and each per_model entry, on both periods. per_backend is a selection-time filter, not a token dimension. The per-key rate_limit_tokens_per_minute and rate_limiting.tier_overrides minute budgets are narrower local overrides applied at claim time, so they keep the value the operator wrote.

A shared Redis bucket already carries the deployment total, so the config hint is inert while a Redis backend is in use and config validate warns when both are configured. A Hub share still applies while present, because an accepted share is never dropped.

Hot reload¶

A changed divisor, or a Hub push that changes a share, rebuilds the budgets and resets the token bucket maps, exactly as a changed tokens_per_minute does. New requests enforce the new capacity from the next window, and a reservation already taken settles on the parameters recorded on its hold rather than on the live ones.

config validate warns when an in-memory deployment carries a nonzero admission-time token budget and no budget_division, because the effective limit is then per replica. The warning names the two alternatives: the config hint, or a Hub enrollment that pushes a share.

Observability¶

Metrics, all labeled by the fixed dimension name (global, client, api_key, model):

• rate_limit_token_budget_tokens_total{dimension, outcome}: tokens with outcome reserved, refunded, or charged, aggregated across both periods.
• rate_limit_token_budget_period_tokens_total{dimension, budget, outcome}: the same movements split by the period that charged them (minute or day, #2327).
• rate_limit_token_budget_refusals_total{dimension}: refusals by the refusing dimension, across both periods.
• rate_limit_token_budget_period_refusals_total{dimension, budget}: refusals split by the period that refused.
• rate_limit_exceeded_total{limit_type="token_budget"}: the aggregate refusal counter shared with the request-count dimensions.
• rate_limit_skipped_total{dimension}: dimensions skipped for a missing identity, under token_budget_<dimension>. A token-bucket map at its entry ceiling fails open and records the same counter under token_budget (minute map) or token_budget_day (day map), so a capacity problem is observable rather than a silent admission (issue #2899).

A configuration reload rebuilds the budgets and clears the token bucket maps, exactly as it does for the request-count dimensions: a changed tokens_per_minute or tokens_per_day gives every bucket a fresh window under the new numbers immediately, and the next request enforces them.

Response headers report only the identity-scoped per_api_key budget: x-ratelimit-tokens-limit, x-ratelimit-tokens-remaining, and x-ratelimit-tokens-reset for the minute bucket, and x-ratelimit-tokens-day-limit, x-ratelimit-tokens-day-remaining, and x-ratelimit-tokens-day-reset for the day bucket. per_client, per_user, per_team, per_model, and global are drawn on by people other than the caller, so their levels are withheld for the same reason the request-count headers withhold shared quotas. The values are the bucket capacity, the tokens available after this request's reservation, and the Unix time at which the bucket is whole again.

The request-count quota keeps its own x-ratelimit-* headers. The token families are separate because the two buckets are independent: a request can be close to its request-count limit while its token budget is nearly empty, or the other way round.

Metric aggregation across replicas¶

Every rate-limit family and the data-plane budget families (router_budget_*, see Budgets) falls into one of four aggregation classes. A per-replica enforcement counter is incremented where this replica makes the decision, so the fleet value is the sum, and rate()/increase() windows absorb restarts and scale events. A process-local gauge describes this process's own memory, so a sum is meaningless. A histogram is aggregated by summing its buckets across replicas, then taking the quantile.

When the bucket itself is shared (storage: redis), rate_limit_tokens_remaining is a shared-state view: every replica observes the same bucket, so summing the series double-counts, and one series or max by (bucket_type) reports the deployment's level. The in-memory backend keeps a bucket per replica, so each series describes only that replica's share.

Alert on an aggregate, not on a single replica's series. A fixed threshold on one series fires spuriously as replicas are added or removed: the same fleet traffic is split across more series, or one series disappears, while the fleet total is unchanged. Sum the counter series before rate(), and sum the histogram buckets before histogram_quantile; never average per-replica quantiles.

# Counter: aggregate the fleet, never one replica's series.
sum by (limit_type) (rate(rate_limit_exceeded_total[5m])) > 1

# Histogram: sum the buckets across replicas, then take the quantile.
histogram_quantile(
  0.95,
  sum by (le, tenant_dimension) (rate(rate_limit_fair_queue_wait_seconds_bucket[5m]))
) > 5

Per-key overrides¶

Decision for issue #2426: accepted and implemented. A key may carry its own burst and minute token budget, settable in the config file and through the Admin API.

• api_keys[].rate_limit_burst replaces the burst capacity of the request rate in effect for the key, which is its own rate_limit when set and rate_limiting.limits.per_api_key otherwise. A bucket smaller than one second's refill is accepted; it still admits the configured rate.
• api_keys[].rate_limit_tokens_per_minute replaces the per_api_key minute budget for that key, and 0 disables it. The overridden budget drives the key's own x-ratelimit-tokens-* headers, and the other admission-time dimensions keep applying.
• A per-key day budget is not part of this override. tokens_per_day stays a dimension-level budget, and a budget shared across all of one user's keys is issue #2381.
• Precedence stays explicit: the key's own rate, then the per_api_key default. The overrides change only the properties named above, so rate_limit: 0 still means no request-rate limiting for the key.

Related designs¶

• Long-window key budgets (#2323, PR #2687): max_tokens_budget over a rolling 1d, 7d, or 30d window is a quota, not a per-minute rate. It is a separate check at the auth seam; both can apply to one key, and a request must pass both. The daily budget documented here is the other shape: a leaky-bucket refill at tokens_per_day / 86400 tokens per second, charged at admission on the router's own dimensions rather than a sliding window over usage history.
• Data-plane budget engine (#2461): extends that rolling-window tracker into named windows and per-key, per-model, and global scopes, with token and micro-USD cost dimensions, hot-reloadable from the budgets: section. max_tokens_budget stays the explicit per-key override and suppresses the per_api_key default for that key; the rate dimensions documented here stay independent, and a request must fit both. See Budgets.
• Monetary pricing feeds the response cost headers and, when a scope sets a cost_usd limit, the budget cost dimension; it is not a per-minute token rate.
• Hub tier minute limits are the control-plane equivalent described above.

Decisions¶

Issue #2898 closed the four questions this record used to carry. Each decision is implemented and pinned by a test.

• No input/output split windows. The local dimension keeps one window per period, and a reservation charges its input estimate and its output limit against that same window. A split shape would be a second, incompatible config surface beside tokens_per_minute and tokens_per_day, and the Hub's input_tpm/output_tpm tier fields already provide per-direction windows on the hub-bound path. Rejected rather than deferred: nothing needs a third local window until a deployment shows the shared window refusing traffic the split would admit. Pinned by a_reservation_charges_input_and_output_against_one_window_per_period.
• The reservation margin is zero beyond the input estimate and the output ceiling. There is no extra percentage and no separate config field. The estimate is already an over-count and the output half reserves the stated ceiling, so a further margin would double-count; reconciliation refunds the unused part, which is where accuracy is recovered. A tunable margin would also require an operator to know the estimator's error distribution. Pinned by the_hold_is_the_estimate_plus_the_output_ceiling.
• The serving request owns the reservation under deduplication. Each request takes its own admission reservation; the leader (the request that dispatched) settles its own, and the waiter, which replays the leader's usage without a backend call, settles nothing and is released in full. No handle is shared, so there is no double settle and no double release. Pinned by a_coalesced_pair_settles_once_and_releases_once.
• A cache hit charges nothing, including its input tokens. The budget meters backend consumption; a response-cache hit makes no backend call, so no tokens were generated or read. The caller's traffic is still counted by the request-count dimensions, which is the right meter for a request the router answered itself. The admission hold is released, never settled. Pinned by a_cache_hit_releases_its_input_and_output_hold.