Rate-Limit and Quota Webhook Notifications¶

The notifications: section pushes the rate-limit and quota refusals the router already makes to application-owned webhooks. The audience is the application behind an API key, not the operator; operator alerting is the separate alerting: section (issue #2356). The feature is opt-in and disabled by default (issue #2416).

Scope¶

When enabled, the router sends a signed, bounded JSON event to each configured target for these events:

• rate_limit_exceeded: a local rate-limit or budget refusal, including the Hub tier minute limits.
• quota_low: the first crossing of quota_low_fraction on a Hub-synced monthly budget.
• quota_exhausted: a spent Hub monthly budget or a spent local key token budget.

Delivery runs entirely off the request path. Producers gate on an atomic enable mask, coalesce repeats, and perform one non-blocking try_send per matching target. No producer awaits and no producer touches the network, so a slow or failing receiver can never delay or fail a request.

With no notifications: section, or with enabled: false, no task is spawned, no event is queued, and no request is made. Enabling the section with no target is allowed and inert, and startup says so once.

Configuration¶

notifications:
  enabled: true
  targets:
    - name: "app-owner"
      url: "https://hooks.example.com/continuum"
      signing_secret: "${WEBHOOK_SIGNING_SECRET}"
      events: [rate_limit_exceeded, quota_low, quota_exhausted]
      keys: []
      timeout: "5s"
      allow_private_network: false
  quota_low_fraction: 0.9
  cooldown: "1m"
  max_queue: 256
  retry:
    max_attempts: 3
    initial_delay: "500ms"
    max_delay: "30s"

The section is validated whether or not it is enabled, so a malformed value fails at load time instead of at enable time. Reload class: immediate for the section and its targets.

The quota_low_fraction value is a fraction, not a percentage: 0.9 means 90 percent of the monthly budget.

Event sources¶

A StoreUnavailable or TrackerFull refusal is an internal failure, not the caller's quota, so it is not a notification event. A 429 proxied from a backend is the provider's quota, not a router event, and is never emitted.

Delivery contract¶

Each target owns a bounded queue (max_queue) and one delivery task. A dead endpoint therefore cannot head-of-line-block another target.

• A producer computes one deduplication key per event and subject, "{event}:{key_id-or-unattributed}". Inside the cooldown window the occurrence is counted in notification_coalesced_total and not sent; the first event after the window carries the suppressed count. The cooldown map holds one entry per event and subject; expired entries are pruned once the map passes a soft size, but there is no hard cap, so its memory follows the number of distinct keys seen inside one window.
• One event is offered to every target that subscribes to its class and whose keys filter matches. An empty filter matches every key; an event with no identified key reaches only targets with an empty filter.
• A full queue drops the event for that target and counts notification_dropped_total{target}. It never blocks the producer.
• Each attempt is bounded by the target timeout. A non-2xx status or a transport error retries with exponential backoff capped by retry.max_delay, up to retry.max_attempts. A 2xx ends the event.
• Settling a reload replaces the target list. Events already queued keep the target that accepted them; the previous tasks drain and exit. On shutdown a task stops starting new deliveries after five seconds; a delivery already in flight may finish after that, bounded by its own timeout and retry budget.
• The event id is generated once and is stable across retries, so a receiver can deduplicate.

Payload¶

{
  "id": "router-1-67c2f0a1-3",
  "event": "rate_limit_exceeded",
  "occurred_at": "2026-11-03T10:00:00Z",
  "router_id": "router-1",
  "subject": {"key_id": "key-42"},
  "detail": {
    "limit_type": "rpm",
    "retry_after_seconds": 30,
    "limit": 600,
    "remaining": 0,
    "window": "minute"
  },
  "suppressed": 0
}

subject.key_id is the key identifier and is omitted when the key is unknown. It is never the key value. detail carries only static event and limit names and bounded numbers; every number is capped at half of u64::MAX, every free-form string is capped at 64 bytes (or replaced with unknown), and the key id is capped at 128 bytes. No API key material, prompt or completion text, hub token, or request header is ever included.

Inside detail, dimension names the local token or in-flight dimension that refused (api_key, global, user, team, ...) when the producer knows it, and window names the budget period (minute, day, monthly). They are separate fields, so window never carries a dimension name.

Signature¶

With signing_secret set, every delivery carries these headers:

Content-Type: application/json
X-Continuum-Event: rate_limit_exceeded
X-Continuum-Delivery: router-1-67c2f0a1-3
X-Continuum-Timestamp: 1793709600
X-Continuum-Signature: sha256=<lowercase hex HMAC-SHA256(secret, "{timestamp}.{body}")>

The body is serialized once to bytes, signed, and sent unchanged, so a receiver verifies exactly what was transmitted. A receiver should:

1. Read X-Continuum-Timestamp and reject a value older than its tolerance window.
2. Recompute HMAC-SHA256 over "{timestamp}.{raw_body}" with the shared secret.
3. Compare the hex digest with the sha256= value in constant time.

import hashlib
import hmac
import time


def verify(secret: bytes, timestamp: str, raw_body: bytes, header: str, tolerance: int) -> bool:
    if abs(int(time.time()) - int(timestamp)) > tolerance:
        return False
    expected = "sha256=" + hmac.new(
        secret, timestamp.encode() + b"." + raw_body, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, header)

Without signing_secret, no timestamp or signature header is sent; the payload is still the bounded envelope above.

Security¶

Targets are validated at configuration load and again when they are applied, including on reload.

• url must use https and must not embed credentials.
• The strict validator rejects loopback, private, link-local, cloud metadata, and other non-public addresses. A host name that does not resolve is a configuration error.
• allow_private_network: true lifts the address blocklist for an in-cluster receiver; the scheme and shape checks still apply.
• The addresses resolved during validation are pinned for the connection, closing the gap between resolution and connection. Redirects are disabled, and each attempt has a timeout.
• The signing secret is a credential. It is masked in configuration export and diff, never logged, and may be written as ${ENV_VAR}. The target URL is masked too, because the path may carry a receiver token.

Metrics¶

Labels are configuration-bounded: event names are a closed set, and target is the validated name, else the URL host, else default.

Exclusions¶

• Per-key self-registration of endpoints through the Admin API or the Hub is not offered. Targets come from the static configuration file, applied at startup or on hot reload.
• The payload does not share the Hub usage-record schema; it is a separate, lighter envelope.
• The local budgets: scope engine has no low-water warning. quota_low is emitted for Hub-synced monthly budgets only, because that is the only place a low-water mark is computed once per window.
• Backend and provider 429s and 5xx responses are not notification events.