Alerting Design¶

This page documents the opt-in outbound alerting feature for issue #2356: push notifications for backend health, circuit, and budget events. It describes the configuration surface, the events the router already produces, and the delivery contract. The implementation lives in src/services/alerting/, with producers in src/health.rs, src/core/circuit_breaker/state.rs, and src/control_plane/policy_store.rs.

Scope¶

Goals:

• Notify an operator when a backend is de-routed, when a circuit opens or closes, and when a key's monthly budget is exhausted or approaches a configured fraction.
• Keep the request path independent of delivery. A slow or unreachable receiver must never add latency to inference.
• Reuse signals the router already computes instead of adding new probes or accounting.

Non-goals:

• Replacing Prometheus, OTLP, the process log, or the Admin API. Alerting is a push convenience beside them, not a new source of truth.
• Routing rules such as per-team channels or escalation. One webhook endpoint receives every enabled event.
• Sending prompt or completion content. Alert payloads carry metadata only.

Configuration¶

The alerting: section is optional and off by default.

alerting:
  enabled: false
  webhook:
    url: "https://hooks.example.com/continuum"
    timeout: 5s
  events:
    backend_unhealthy: true
    backend_recovered: true
    circuit_opened: true
    circuit_closed: true
    budget_exhausted: true
    budget_fraction: 0.9
  cooldown: 5m
  max_queue: 256
  retry:
    max_attempts: 3
    initial_delay: 500ms
    max_delay: 30s

Secrets in webhook.headers (a follow-up field) use ${ENV_VAR} substitution and are masked in the Admin API and config export, like other credential fields.

Event Sources¶

Without the control-plane feature there is no local spend authority, so the two budget events never fire. The startup log states that the budget events are inert, in line with the project rule that config types are always compiled while only the runtime is gated.

Delivery Contract¶

• Single task: one background task owns the queue and performs every HTTP POST. Event producers call a non-blocking try_send and continue.
• Bounded queue: at max_queue entries try_send fails, the new event is dropped, and alert_dropped_total is incremented. Request handling is never delayed by a full queue.
• Retries: failures other than a 2xx response retry with exponential backoff bounded by retry.max_delay. A 2xx ends the event. Exhausted retries increment alert_deliveries_total{result="failed"}.
• Cooldown: the notifier keeps one timestamp per event and subject (backend name or key id). Occurrences inside cooldown are counted; when the window closes, a summary carries the suppressed count.
• Reload: the section is re-read on hot reload. A new URL takes effect for events queued after the reload; queued events keep the URL they were formed with.
• Shutdown: after the shutdown signal the delivery task drains queued events for up to a hard-coded 5 seconds, and the server does not await it, so events still pending when the process exits are dropped.

Payload¶

Every request is POST with Content-Type: application/json.

{
  "event": "circuit_opened",
  "occurred_at": "2026-10-28T10:00:00Z",
  "router_id": "continuum-router-1",
  "subject": { "backend": "openai-primary" },
  "detail": { "previous_state": "closed", "state": "open", "failure_status_code": 503 },
  "suppressed": 0
}

• event is one of the enabled event names above.
• router_id is the control-plane router id when the router is enrolled with a Hub, otherwise the host name.
• subject names the backend or the key id, never the key value.
• detail carries only fields the source event already has, such as the provider status code or the exhausted budget window.
• suppressed counts occurrences coalesced by the cooldown.

Security¶

• The URL must use https and passes the strict SSRF validator (src/infrastructure/common/url_validator.rs), which refuses loopback, private, link-local, and cloud metadata addresses, including host names that resolve to them. webhook.allow_private_network: true (a new field, false by default) lifts the address check for an in-cluster receiver. Validation alone does not close the gap between resolution and connection, so the delivery client pins the validated addresses.
• A URL with embedded credentials is a load error; credentials belong in webhook.headers through environment substitution.
• Payloads never include prompt text, completion text, API keys, or hub tokens.
• Receiver errors are logged without the URL and without request headers.

Metrics¶

Testing Plan¶

• Unit tests for payload serialization, cooldown accounting, and queue shedding.
• A test with a local HTTP server that asserts retry on a 500, success on a later 2xx, exhausted retries counting a failure, and that enqueueing never touches the network.
• Config tests for validation: a missing URL when enabled, a non-https URL, embedded credentials, and an out-of-range fraction are load errors.
• The configuration-assistant drift test keeps the sections_count and ### alerting catalog entry in step with the schema; keep this page and the alerting: block in config.yaml.example in step when the section changes.

Relation to Client Notifications (#2416)¶

The rate-limit and quota webhook design (Webhook Notifications: Design Record, issue #2416) pushes events to the client application that owns a key. Alerting pushes operational events to the operator. The two sections stay separate because their receivers, subscriptions, and payloads differ, but they should share one delivery implementation (bounded queue, retry with backoff, SSRF validation, delivery metrics) so the guarantees match.

Rollout¶

The implementation landed in the order below.

1. Config types and validation, inert until enabled is true.
2. The notifier task and the health and circuit events.
3. The budget events with the control-plane gate.
4. The config.yaml.example section and the configuration-assistant entry, in the same change.