Budgets¶

The budgets: section enforces long-window quotas on tokens and USD cost in the data plane, without a Continuum Hub. It answers a different question from rate limiting: rate limiting bounds how fast a caller may spend, and budgets bound how much it may spend before a rolling window clears.

A router with no budgets: section, or with enabled: false, is unchanged: no counter is created, no request is charged, and no budget header is emitted.

How the token limits coexist¶

Three settings can bound a key's tokens. They are independent checks, and a request must pass every one that applies to it.

max_tokens_budget keeps its original meaning: the explicit per-key rolling-window token quota of issue #2323, read and charged on the auth seam. It is also the per-key override of this section. When a key sets it, the budgets.scopes.per_api_key default is not admitted and not charged for that key, so one key is never charged for the same scope twice. A key without it follows the fleet default. budgets.scopes.per_model and budgets.scopes.global still apply to that key, because those scopes are shared and not the key's own override.

tokens_per_minute and tokens_per_day are rates, not quotas. A rate refills continuously and a tenant may spend its whole minute or day at once; a budget ages out over a rolling window. Both checks run at admission, and neither replaces the other.

Configuration¶

budgets:
  enabled: true
  storage: memory            # memory | redis
  fail_open: true            # redis outage: admit and count best-effort
  windows:
    - id: daily
      period: 24h
    - id: monthly
      period: 30d
  scopes:
    per_api_key:             # default for keys without max_tokens_budget
      tokens: 2000000
      cost_usd: 50.0
    per_model:               # keyed by the client-facing model name
      gpt-5.6-sol:
        tokens: 500000
    global:                  # one fleet-wide bucket
      tokens: 20000000
      cost_usd: 500.0
  cost_source: model_metadata

A limits object sets tokens, cost_usd, or both. At least one must be set for the scope to do anything, and each configured value must be greater than zero.

config validate refuses duplicate window ids, an unparseable period, an enabled section with limits but no windows, a zero token limit, a non-positive or non-finite cost, and an empty model name.

Windows¶

A window is a fixed-length rolling period, not a calendar boundary. Usage falls out one time slice at a time, so a 24h window is always the last 24 hours. calendar:day and calendar:month are aliases for 24 hours and 30 days, not resets at midnight or on the first of the month.

A scope's tokens and cost_usd apply to every configured window. A request must fit every window of every applicable scope. In the example above, a key that is under its daily limit is still refused when it would cross the monthly one, and a model bucket and the global bucket are checked on the same request.

Scopes and precedence¶

• Per-key: the caller's own identity. It is the only scope whose level is reported. A key with max_tokens_budget does not use this scope.
• Per-model: keyed by the client-facing model name. A request for a model with no entry skips the scope. Every caller of that model shares the bucket.
• Global: one bucket for every caller and model.

The checks run in that order for each window, and the first scope that cannot cover the request refuses. A refusal names that scope, the window, and the configured limit. It never reports the current usage of a shared scope.

The admission estimate is the request's input estimate plus its output limit, at the model's published prices. The estimate is held until the response body ends. Completed usage is recorded once from the usage seam, for buffered and streamed replies alike.

Cost¶

Cost comes from model-metadata.yaml pricing, resolved through the same PricingInfo path as the response cost headers. It is stored as integer micro-USD so a long window cannot accumulate float drift.

A model without pricing still counts its tokens but adds no cost. A limit that sets only cost_usd therefore does not bound an unpriced model.

Storage¶

storage: memory is the default. Counters live in the process, so behind a load balancer each replica enforces the quota against the traffic it serves. Use storage: redis when the fleet must share one quota.

Redis storage reuses the rate_limiting.redis endpoint and key_prefix; there is no separate endpoint under budgets:. The redis-cache Cargo feature is required at build time. Counter keys live at least as long as their longest configured window even when rate_limiting.redis.ttl is shorter, so a monthly budget is not forgotten after the cache TTL. When rate_limiting.redis is not configured, or the store cannot be built at startup, the router logs a warning and enforces the section with per-process memory counters.

On a Redis outage:

• fail_open: true (the default) admits the request and counts it on the replica's local tracker. The shared quota is temporarily under-enforced, and the outage counts on router_budget_storage_unavailable_total{action="fallback"}.
• fail_open: false refuses with 503 and Retry-After: 5 until Redis answers, counting action="refused". The shared quota stays exact at the cost of availability.

fail_open and the endpoint are read at startup and on reload; the Redis store is built from the startup configuration, so changing the endpoint requires a restart.

Refusals¶

An exhausted budget is refused with 429 Too Many Requests and Retry-After set to the refusing window in seconds.

• OpenAI ingress (/v1/chat/completions, /v1/completions, /v1/responses): the standard error envelope, type: budget_exceeded, code: budget_exceeded, and error.details carrying scope, window, unit, and the configured limit (or limit_micro_usd). A model refusal adds model.
• Anthropic ingress (/anthropic/v1/messages, /v1/messages): the Anthropic envelope, {"type":"error","error":{"type":"rate_limit_error",...}}, with the same error.details sibling.

A full process-local tracker is refused the same way rather than admitting an untracked key.

Response headers¶

An admitted request whose key scope reserved reports the key's own level:

The per-model and global levels are deliberately withheld. Those buckets are shared by other callers, so publishing their fill level would leak how much of a shared quota another tenant has consumed. The same rule applies to the shared rate-limit dimensions.

The headers are absent when the per-key scope has no token limit, when the request qualified for no scope, or when budgets are off. A cost-only per-key limit reports no level rather than inventing a token number.

Admin API¶

GET /admin/budgets returns the effective configuration and the current counters of every scope and window. POST /admin/budgets/reset clears a scope's recorded usage.

curl -H "Authorization: Bearer $ADMIN_TOKEN" http://localhost:8080/admin/budgets

curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"scope": "per_api_key", "window": "daily"}' \
  http://localhost:8080/admin/budgets/reset

The reset body takes scope (per_api_key, per_model, or global), and optionally window, api_key_id (per-key only), and model (per-model only). Clearing a Redis-backed scope deletes the shared counters, so every replica sees the reset. Both endpoints sit behind the Admin auth layer.

Metrics¶

The label sets are closed by construction. No model name, key id, or other free text is carried.

Hot reload¶

The budgets: section is read from the live configuration on every request. Changing a limit, window, scope, fail_open, or storage applies to the next request; a widened limit takes effect immediately.

The admission layer is mounted from the startup configuration. A router that starts with a budgets: section, or with rate_limiting enabled, picks up later section changes on reload, including enabling the section. A router that starts with neither has no admission layer to enable, and adding budgets: to such a deployment requires a restart.

Limitations¶

• Memory counters are per process. A fleet of N replicas enforces up to N times a memory quota.
• Only model_metadata pricing is supported. A model with no pricing contributes tokens but no cost.
• A budget refusal stops the request before dispatch. It does not reserve a backend slot, and a canceled client releases its estimate.
• Admin-created runtime keys have no max_tokens_budget; they follow budgets.scopes.per_api_key.

See also¶

• Rate limiting and token budgets for the rate shape and its design record.
• Security & Admin for max_tokens_budget.