Metrics and Monitoring¶

This document describes the metrics and monitoring capabilities of the Continuum Router.

Table of Contents¶

• Overview
• Quick Start
• Configuration
• Available Metrics
• Integration
• Grafana Dashboard
• Alerting
• Examples
• Best Practices

Overview¶

The Continuum Router exposes Prometheus-compatible metrics for monitoring system health, performance, and usage patterns. The metrics system is designed to be:

• Lightweight: Minimal performance overhead
• Broad coverage: Covers HTTP, backend, routing, model, streaming, and cache subsystems
• Production-ready: Includes cardinality limits and proper labeling
• Easy to integrate: Works with standard Prometheus/Grafana setups

For restart-survival history without standing up a full Prometheus stack, see the Persistent Metrics Log. It snapshots the registry to a local SQLite store and exposes recent history via GET /admin/metrics/history.

Metrics answer how much and how often. For per-request causality, which backend was chosen, how many attempts a request took, and why each attempt ended the way it did, see Distributed Trace Export, which ships the router's own spans to an OpenTelemetry collector over OTLP.

Quick Start¶

1. Enable Metrics¶

Metrics are enabled by default, but /metrics returns 401 Unauthorized until metrics.auth is configured. Use a bearer token for Prometheus:

metrics:
  auth:
    method: bearer
    bearer_token: "${METRICS_TOKEN}"

# View metrics
curl -H "Authorization: Bearer $METRICS_TOKEN" http://localhost:9090/metrics

2. Configure Prometheus¶

Add the router as a target in your prometheus.yml:

scrape_configs:
  - job_name: 'continuum-router'
    static_configs:
      - targets: ['localhost:9090']
    scrape_interval: 15s
    authorization:
      credentials: '<metrics-token>'

3. Import Grafana Dashboard¶

Import the provided dashboard from monitoring/grafana/dashboards/router-overview.json.

Configuration¶

Metrics configuration is done through the main config file:

metrics:
  enabled: true
  port: 9090            # dedicated metrics listener; 0 turns it off
  # bind_address: 0.0.0.0  # default: the host of the first TCP server.bind_address
  path: /metrics
  auth:
    method: bearer
    bearer_token: "${METRICS_TOKEN}"
  max_model_labels: 1000
  # External Prometheus is the durable store for this profile.
  persistence:
    enabled: false

• max_model_labels: Caps the distinct models tracked on model_usage_total, model_tokens_processed, and llm_tokens_total (default 1000, which is also the maximum; minimum 10). Applied at startup, so changes require a restart. Models past the cap are not recorded on those families, and lowering the cap does not evict models that are already tracked.
• max_backend_labels and cardinality_limits (deprecated): Parsed for backward compatibility and ignored (continuum-router config validate warns when either is set to a non-default value). cardinality_limits.max_models duplicates max_model_labels, its sampling fields were never implemented, and the backend label cap is fixed at 1000 because backends can be registered at runtime.
• latency_buckets (deprecated): Parsed for backward compatibility and ignored (continuum-router config validate warns when it differs from the default). http_request_duration_seconds is registered once per process with a fixed bucket layout (0.01 to 1200 seconds) that dashboards and le selectors rely on, so the buckets cannot be changed in configuration.

Omitting auth fails closed. To expose an intentionally unauthenticated endpoint on a trusted network, set auth.method: none explicitly. The startup log marks that choice as UNPROTECTED and names the authenticated alternatives.

A config without a metrics: section uses the defaults: the endpoint is served at /metrics on the main listener and on the dedicated metrics listener at port 9090 (fail-closed until auth is set; if the port is taken, the router logs a warning and starts without that listener), request metrics are recorded, and the persistent metrics log stays off. To turn metrics off, set metrics.enabled: false.

Dedicated Metrics Listener¶

With metrics enabled, the router binds a second TCP listener on metrics.port (default 9090) that serves only the metrics route, behind the same metrics.auth as the main listener. API, admin, health, and WebUI routes answer 404 there, so a network policy can admit the monitoring namespace to this port without exposing the API. The shipped Kubernetes manifests and Helm chart scrape this port.

• Host: metrics.bind_address (an IP address or localhost, without a port) when set, otherwise the host of the first TCP entry in server.bind_address. When server.bind_address has only unix: entries and metrics.bind_address is unset, the listener is not started and the router logs a warning (config validate reports it too), so no TCP port is opened that you did not configure.
• Bind failures: what happens when the port cannot be bound depends on whether you asked for the listener:
  • metrics.port or metrics.bind_address is set: startup fails with an error naming metrics.port and the address, so a deployment that scrapes this port does not come up unmonitored. The shipped Helm values and Kubernetes manifests set port: 9090, as does the production-ha config template. A file saved from config show or the Admin API export of an earlier release carries port: 9090, which counts as set.
  • Neither is set: the router logs a warning with the address and the OS error, then starts without the listener. /metrics stays on the main listener. Set metrics.port to make this failure fatal, or metrics.port: 0 to turn the listener off and silence the warning.
• Port conflicts: when metrics.port equals the port of a main TCP listener, no second listener is started, because /metrics is already served there. In AppProxy ROUTER mode, an appproxy_router.traffic_port equal to metrics.port is refused at startup.
• Turning it off: set metrics.port: 0 to serve metrics only on the main listener, as releases before this change did.
• Main listener: /metrics stays on the main listener as well, for existing scrape configurations. Its rate-limit exemption is unchanged. The dedicated listener has no rate limiter; protect it with metrics.auth and network policy.
• Lifecycle: the listener starts with the server and drains with the API listeners on shutdown. metrics.enabled, metrics.port, and metrics.bind_address are read once at startup; a hot reload that changes them logs that a restart is required.

Environment Variables¶

You can also configure metrics using environment variables:

# Enable/disable metrics
METRICS_ENABLED=true

# Change metrics endpoint
METRICS_ENDPOINT=/custom/metrics

# Enable optional metrics
METRICS_ENABLE_BODY_SIZE=true

Available Metrics¶

HTTP Metrics¶

http_request_duration_seconds uses the buckets 0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300, 600, and 1200 seconds. The buckets above 10 seconds resolve long model calls instead of counting them only in +Inf, and the top bucket matches the 20 minutes of continuum_backend_response_wait_seconds. The duration ends when the handler returns response headers, so for a streaming response it excludes the time spent streaming the body.

The endpoint label keeps the first path segment, or the first two under /v1/ and /admin/. The Anthropic Messages API is served at both /anthropic/v1/messages and /v1/messages, and both mount points (including count_tokens) report endpoint="/v1/messages".

The HTTP middleware reports backend="unknown" because backend selection happens inside handlers and not every route selects a backend. Per-backend traffic and load are available through the backend and prefix-routing metric families; client-supplied headers are never trusted for metric attribution.

Error and Retry Metrics¶

Circuit Breaker Metrics¶

Populated by ordinary proxy traffic when the metrics feature and a circuit_breaker are both enabled.

error_type is slow for a success that crossed a circuit_breaker.latency_as_failure threshold, and the corresponding transition carries reason="slow". Other reason values are bounded: failure_threshold, failure_rate, timeout, probe, and manual.

Request Parameter Policy Metrics¶

Both families use closed, bounded labels. source is one of none, local, hub, or intersection; neither family exports request values, model ids, tier ids, key ids, or policy cursors.

Backend Metrics¶

Cached prompt tokens and TTFT per backend¶

These two families make KV offloading visible per backend. Both record on every build that has the metrics feature, including the default release build; they do not depend on the Continuum Hub agent.

continuum_backend_cached_prompt_tokens_total adds the cached prompt token count the backend itself reported for a request it served: usage.prompt_tokens_details.cached_tokens on OpenAI-compatible chat and completions, usage.input_tokens_details.cached_tokens on the Responses API, and usage.cache_read_input_tokens on Anthropic. Cache-creation tokens are not added. A request counts once, whether it was buffered or streamed; local response-cache replays, failed requests, and requests without a usage block add nothing.

continuum_backend_ttft_seconds starts at provider dispatch, so local routing, request transformation, and guardrail preflight are excluded. The source label keeps two different measurements apart:

• streamed_first_token: dispatch to the first streamed frame of a streaming response. This is the series to compare cold, local-hit, and external-hit requests.
• response_headers: dispatch to the upstream response headers of a non-streaming call. Most engines answer only after generating the whole reply, so this approximates full upstream latency, which is also the first moment a non-streaming client sees any token.

The buckets (25 ms to 60 s) are spaced so that a local GPU prefix hit (tens of milliseconds), a KV connector hit that reloads the prefix from CPU or disk (about 0.1 s to 2 s for long contexts), and a cold long-context prefill (seconds to tens of seconds) fall into different buckets.

TTFT is recorded only where the router observed both instants; an unobserved instant is never substituted. A failed attempt (a dispatch or hop whose usage record's outcome is not completed, length, or usage_missing) takes no response_headers sample and its usage record carries no ttft_ms; a stream's streamed_first_token sample is unaffected by how the stream ended. Coverage by path:

The backend label is sanitized and admitted through the same cardinality guard as the other request-path backend series, and the /metrics scrape drops the series of any backend that is no longer in the pool, so the label set stays bounded by the configured backends.

# Cached prompt tokens per second, per backend
sum by (backend) (rate(continuum_backend_cached_prompt_tokens_total[5m]))

# Streamed TTFT p50 and p95 per backend
histogram_quantile(0.5, sum by (backend, le) (rate(continuum_backend_ttft_seconds_bucket{source="streamed_first_token"}[5m])))
histogram_quantile(0.95, sum by (backend, le) (rate(continuum_backend_ttft_seconds_bucket{source="streamed_first_token"}[5m])))

Request priority and TTFT per priority class¶

When multiple Hub tiers with different request_priority floors share one vllm backend running --scheduling-policy priority, continuum_backend_ttft_seconds{backend,source} aggregates their latency together. continuum_request_priority_requests_total{priority,status_class} and continuum_request_priority_ttft_seconds{priority} split request volume and router-observed TTFT by the effective priority that governed the request:

• priority is the effective integer priority formatted in base 10 (for example "-10" for an interactive tier and "10" for a batch tier), "unmanaged" when no priority was in effect (no tier request_priority, or a hop onto a non-vllm backend where priority enforcement was skipped), or "other" once 16 distinct integer priority labels have already been admitted ("unmanaged" does not consume any of the 16 dynamic slots).
• status_class on continuum_request_priority_requests_total is one of 2xx, 4xx, 5xx, or error.
• continuum_request_priority_ttft_seconds reuses the exact bucket layout (25 ms to 60 s) and observation rules of continuum_backend_ttft_seconds: a sample is taken at the same completion and stream-end seams only when continuum_backend_ttft_seconds takes one.

# Request rate per priority class and status class
sum by (priority, status_class) (rate(continuum_request_priority_requests_total[5m]))

# TTFT p50 and p95 per priority class
histogram_quantile(0.5, sum by (le, priority) (rate(continuum_request_priority_ttft_seconds_bucket[5m])))
histogram_quantile(0.95, sum by (le, priority) (rate(continuum_request_priority_ttft_seconds_bucket[5m])))

Grafana panel JSON (monitoring/grafana/dashboards/router-overview.json):

{
  "datasource": "Prometheus",
  "description": "Router-observed time to first token per effective vLLM priority class (continuum_request_priority_ttft_seconds). Compare interactive (for example -10) and batch (10) tiers sharing a vLLM backend with --scheduling-policy priority.",
  "fieldConfig": {
    "defaults": {
      "unit": "s"
    },
    "overrides": []
  },
  "gridPos": {
    "h": 8,
    "w": 12,
    "x": 0,
    "y": 52
  },
  "id": 16,
  "targets": [
    {
      "expr": "histogram_quantile(0.5, sum by (le, priority) (rate(continuum_request_priority_ttft_seconds_bucket[5m])))",
      "legendFormat": "p50 priority={{priority}}",
      "refId": "A"
    },
    {
      "expr": "histogram_quantile(0.95, sum by (le, priority) (rate(continuum_request_priority_ttft_seconds_bucket[5m])))",
      "legendFormat": "p95 priority={{priority}}",
      "refId": "B"
    }
  ],
  "title": "TTFT by Request Priority Class (p50 / p95)",
  "type": "timeseries"
}

HTTP protocol and stream-limit waits¶

These two families make an HTTP/2 stream-limit stall visible without packet captures. Over HTTP/2 the router multiplexes every request to a host onto one connection, and a request above the server's SETTINGS_MAX_CONCURRENT_STREAMS waits inside the HTTP client until a stream closes. The HTTP client does not report that wait, so both series are router-side observations. The fix for a backend that shows the pattern is http_version: http1.

• continuum_backend_http_in_flight is the backend pool's in-flight count (the same number as backend_current_load), labelled with the backend's configured http_version. On an auto or http2 backend, a value that plateaus at a round number such as 100, 128, or 250 while traffic keeps arriving is the server's stream limit.
• continuum_backend_response_wait_seconds runs from handing a request to the HTTP client until its send settles on the chat and completions dispatch paths (non-streaming chat, completions, and embeddings, plus streaming chat completions including fallback hops and mid-stream fallback). Its buckets reach 20 minutes so a request parked behind a full connection is resolved rather than lumped into +Inf. Each attempt is recorded once, including attempts that fail or are abandoned before any header arrives. The outcome label indicates how the send settled:
  • response: response headers arrived (any HTTP status).
  • error: transport error or timeout.
  • cancelled: the request was abandoned first, typically because the client disconnected while waiting for an HTTP/2 stream slot.

The backend label follows the same sanitizing and cardinality guard as the other per-backend series, and the scrape drops the series of backends no longer in the pool. http_version and outcome are closed three-value sets. The in-flight gauge is set, not accumulated, on each scrape, so concurrent scrapes read consistent values, and a series whose backend left the pool or changed http_version is removed.

A warn log line accompanies the pattern: at most once a minute per backend, when a backend that can negotiate HTTP/2 (http2, or auto with an https:// URL) reaches 100 requests in flight, the router names http_version: http1 as the way out. An auto backend on a cleartext http:// URL always speaks HTTP/1.1, so it never warns.

# Backends whose in-flight count is flat at a stream limit
max_over_time(continuum_backend_http_in_flight{http_version!="http1"}[15m])

# 95th percentile wait for response headers, per backend and protocol
histogram_quantile(0.95, sum by (backend, http_version, le) (rate(continuum_backend_response_wait_seconds_bucket{outcome="response"}[5m])))

# Requests per second abandoned while waiting for headers
sum by (backend) (rate(continuum_backend_response_wait_seconds_count{outcome="cancelled"}[5m]))

# Requests per second that waited more than 60 s for headers
sum by (backend) (rate(continuum_backend_response_wait_seconds_bucket{le="+Inf"}[5m]))
  - sum by (backend) (rate(continuum_backend_response_wait_seconds_bucket{le="60"}[5m]))

Engine Statistics Metrics¶

Emitted only with engine_stats.enabled: true, and only for engine-stats-capable backend types (vllm, sglang, llamacpp, mlxcel, ollama, lmstudio). The poller reads each engine's own load or metrics endpoint (vLLM GET /metrics, SGLang GET /v1/loads with /get_load and /metrics fallbacks, llama.cpp GET /slots and GET /metrics selected via the /props capability booleans, Ollama GET /api/ps, LM Studio GET /api/v0/models) and normalizes the answer into one snapshot per backend.

Three representation rules keep the series honest:

• Absent, not zero. A series for a figure the engine has no concept of is not emitted at all, decided statically per adapter. The one exception is SGLang /v1/loads, whose wire format omits zero-valued fields (msgspec omit_defaults); the adapter reads a missing number there as a truthful 0, so an idle SGLang backend keeps reporting backend_engine_requests_running 0 instead of vanishing.
• A failing scrape makes one statement. On any scrape error the backend emits only backend_engine_stats_scrape_success 0; the other series disappear until a scrape succeeds, so a broken scrape can never be read as an idle engine. The router keeps the last snapshot internally (visible with growing staleness_seconds on the Admin API).
• Multi-rank aggregation. An SGLang data-parallel engine reports one entry per DP rank; ranks merge into one snapshot by summing counts, capacities, and throughput and taking the maximum of usage fractions and hit rates (the saturated rank is the one that matters). The same rule applies when a Prometheus body carries multiple label sets for one series.

The two *_tokens_total counters apply reset detection (an engine value below the last observed one counts as a restart, and the new value is the delta since the restart). Reset detection is applied only to series the engine declares as counters; llama.cpp's /metrics mixes counters with gauges, and a gauge legitimately decreases.

Local versus external prefix-cache hits (vLLM)¶

vLLM exports two prefix-cache counter pairs, both counted in tokens, and the adapter reads each into its own cumulative ratio:

• Local (backend_engine_prefix_cache_hit_ratio): vllm:prefix_cache_hits / vllm:prefix_cache_queries, the engine's own GPU-resident prefix cache.
• External (backend_engine_external_prefix_cache_hit_ratio): vllm:external_prefix_cache_hits / vllm:external_prefix_cache_queries, the tokens a KV connector such as LMCache served from its offload tier out of the tokens vLLM asked it for. A hit here is a prefix the GPU cache had evicted or never held and the connector reloaded instead of the engine recomputing it.

The external series exists only when the engine exports the connector counters, which vLLM does when it runs with a KV connector, and only once the connector has been queried; an engine without a connector publishes no external series rather than a 0. SGLang has no equivalent query/hit counter pair (its HiCache exports sglang:cached_tokens_total by cache_source instead), so the external gauge is vLLM-only; scrape SGLang's own /metrics for its host and storage tier hits.

A backend removed by hot reload loses its backend_engine_* series on the next scrape; a backend added by hot reload starts reporting within one polling interval. See engine_stats in config.yaml.example for the configuration surface, and GET /admin/backends/{name}/engine-stats for the JSON snapshot.

An engine's answer is treated as untrusted input, because a serving engine can be buggy or compromised independently of the router:

• Confined reach. The poller fetches only the configured backend URL, or a metrics_url that resolves to the same host, and it follows no redirects. The same-host rule is checked at config load and again immediately before each fetch, and it fails closed when the backend URL does not parse. A metrics_url may not downgrade an https backend to plaintext http, because the backend API key is sent with the metrics fetch; allow_external_metrics_url widens the host rule and does not lift that one.
• Nothing from the body reaches a log. Parse failures report the structural classification and the position, never the offending value, so no slice of an engine body (notably the llama.cpp /slots body) can reach the router log through a failed scrape. Transport failures report a credential-free URL.
• Bounded intake. The response is read against max_body_bytes chunk by chunk, the Prometheus parser caps samples per body and labels per sample, and the only engine free text a snapshot keeps (model names and the engine version) is length-bounded and stripped of invisible control characters. Numeric aggregation saturates instead of overflowing.

Usage Reconciliation Metrics¶

Emitted only while usage.reconciliation is configured and enabled. The reconciler compares, per backend and per window, the output tokens the usage ledger recorded with the delta of the engine's generation counter (backend_engine_generation_tokens_total above reads the same counter), so these series are the proof that the ledger counts what the engines generated. Every family is keyed by the same sanitized backend label as the engine family; a backend that leaves the reconciled set (backends[].reconciliation: false, removed by hot reload, or the block disabled) loses its series at once.

The two token counters do not match one for one: ledger tokens recorded before the first engine baseline are counted on the ledger side only, since the baseline already contains that generation. Compare windows with usage_reconciliation_drift_ratio or GET /admin/usage/reconciliation, whose totals cover only compared windows, reset windows included: a reset window counts the tokens generated between the last scrape before the engine restart and the restart on the ledger side only, so the totals drift negative by that amount after each restart. Types without a generation counter (anything but vllm, sglang, llamacpp, mlxcel) count only no_engine_data windows.

Prompt Logging Metrics¶

Emitted while prompt_logging.enabled is true. The prompt logging guide explains how to read them.

Model Service Metrics¶

• model_background_refreshes_total vs. completion counters: A pass superseded while in flight is retried in place and each retry increments model_background_refreshes_total again, whereas model_background_refresh_successes_total and model_background_refresh_failures_total advance once per revalidation.
• model_superseded_aggregations_total: Includes membership moves detected by the post-store re-check. Each supersession triggers an immediate re-aggregation, except after three consecutive supersessions, when the next revalidation waits a 2 s re-arm interval.

Diagnosing a slow model list¶

model_refresh_duration_seconds tells you that an aggregation was slow; model_backend_fetch_duration_seconds tells you which backend made it slow, because the model-list fan-out is concurrent and the total is therefore the slowest single backend.

# Which backend dominates the refresh, 95th percentile over 5 minutes
histogram_quantile(0.95, sum by (backend, le) (rate(model_backend_fetch_duration_seconds_bucket[5m])))

Both histograms bucket up to 60 seconds. A backend exceeding the per-attempt request_timeout also emits one WARN log line naming the backend, its duration, and its attempt count, so the same finding is available without Prometheus and without raising the global log level to DEBUG:

WARN Slow model fetch from backend backend="claude-bedrock" duration_ms=15503 attempts=3 threshold_ms=5000 outcome="error"

model_backend_fetch_duration_seconds carries one series per backend name. Backend names are operator-defined and bounded by the configuration, so cardinality is normally small; a deployment that registers backends dynamically (AppProxy replicas) should expect one series per distinct replica name.

Routing telemetry is exported by the live KV cache and smart routing families below. Unsupported members of the former generic routing and model-service collectors are not registered.

Active streaming response bodies contribute to http_active_connections. The router does not currently export generic streaming duration or post-response error metrics; mid-stream fallback has the dedicated live metrics below.

Mid-Stream Fallback Metrics¶

These metrics are emitted when the mid-stream fallback feature is enabled (streaming.mid_stream_fallback.enabled: true).

Reason Label Values for streaming_fallback_total¶

Key PromQL Queries¶

# Mid-stream fallback rate
rate(streaming_fallback_total[5m])

# Fallback recovery success rate
sum(rate(streaming_fallback_success_total[5m])) /
sum(rate(streaming_fallback_total[5m]))

# Median accumulated tokens at fallback trigger
histogram_quantile(0.5, rate(streaming_fallback_accumulated_tokens_bucket[5m]))

Responses Bridge Metrics¶

Counts /v1/chat/completions requests the router dispatched to the upstream /v1/responses endpoint instead. Always registered; the counter simply stays at zero when nothing is bridged.

Reason Label Values for responses_bridge_total¶

The two mean different things operationally. responses_only_model is a standing property of the deployment and its rate tracks traffic to those models. tools_with_reasoning is a property of the traffic itself: it tells you how much of your agent traffic is paying the translation round-trip, and it drops to zero on its own once OpenAI lifts the restriction and the metadata flag is cleared.

# Share of bridged traffic that is the conditional tools bridge
sum(rate(responses_bridge_total{reason="tools_with_reasoning"}[5m])) /
sum(rate(responses_bridge_total[5m]))

Tool Conversion Metrics¶

The Anthropic and Gemini request conversions forward only function tools; the Gemini conversion also maps web_search, web_search_preview, code_interpreter, and Gemini's own google_search and code_execution to its built-ins. Any other declared tool type is removed from the upstream request, logged at warn with its type, and counted here. The client is not told; watch this counter when a client relies on OpenAI built-in tools against Anthropic or Gemini backends.

target is anthropic or gemini. tool_type is the tool's type when it is one of web_search, web_search_preview, file_search, code_interpreter, computer_use_preview, image_generation, mcp, local_shell, or custom, and other for any other value.

tool_choice_unenforced_total counts a Gemini native tool_choice the router validated but could not fully apply. reason is forced_without_functions when a required or named choice arrived without function declarations, and none_with_built_ins when built-in tools had to be removed from a none request. Both outcomes still answer the client normally, so this counter and its warn log are the only signal.

Model Experiment Metrics¶

Per-variant attribution for model experiments. A dedicated family, so existing request, latency, and token series keep their label sets. Label values come from model_experiments configuration (at most 32 experiments with 16 variants each) or from closed sets.

Shadow Traffic Metrics¶

Metrics for shadow traffic mirroring. They are registered at startup, so a router that never enables the section still reports them at zero. model and backend come from the shadow configuration and the selected backend, so cardinality is bounded by configuration rather than by traffic.

Fallback Metrics¶

usage_quality_judgments_total adds up to all eligible requests across these outcome values:

• attached: The judgment was sealed into the push batch carrying the usage record (not subtracted if the Hub later refuses the whole batch).
• late: Ready after the record's push batch formed, or after the record was dropped (late judgments represent the slowest evaluations, not a random sample).
• sampled_out: Excluded by sampling.
• skipped: Nothing to judge, a replaced or blocked reply, no usage record emitted, or the request ended before a reply was served. Also skipped for:
  • Streamed replies (stream="true") when an output guardrail cut or rewrote the stream, mid-stream fallback spliced two generations, the stream carried an error, or the client disconnected.
  • /anthropic/v1/messages replies containing a tool_use or server_tool_use block, or ending with pause_turn.
  • /v1/responses and responses_only bridge replies whose input has no user turn, that contain a client-executed call item, or that ended as failed or cancelled.
• shed: Skipped because max_in_flight was full.
• failed: Decision call failed or timed out.
• invalid: Missing or non-finite probability, or a judgment rejected by the protocol.
• unresolved_model: Decision response named no model.

usage_tokenize_corrections_total records one of the following result values (the record's outcome never changes, so a usage_missing stream still counts in usage_missing_total):

• applied: The engine's /tokenize count replaced the router's (token_source="engine_tokenize").
• timeout: timeout_ms elapsed first; the router count stands.
• unsupported: Backend type has no engine tokenizer route, its URL is not http(s), or it left the configuration.
• skipped_truncated: Retained text hit its 100,000-byte cap.
• shed: max_concurrent was full.
• error: Transport failure, non-2xx status, response over 1 MiB or unreadable, or an implausible count (0 for non-empty text, or greater than the UTF-8 byte length). The router count stands and one WARN log line names the backend, engine, and failure kind.

The backend label of fallback_attempts_total is the backend the failed attempt dialed, and of fallback_success_total the backend the serving hop dialed. The value not_dispatched means the chain advanced with no dial at all, for example when every backend of the model was unhealthy; unknown is never written.

The original_model and fallback_model labels carry the chain's model ids as configured, / and : included, so meta-llama/Llama-3.1-8B-Instruct and meta-llama_Llama-3.1-8B-Instruct stay two series. A value is truncated at 128 bytes on a character boundary, and any character outside the chain alphabet (whitespace, a control character, other punctuation) is exported as _. The series count is bounded by configuration: these families are recorded only for a model that has a chain and for the hops it names, and a fallback.default_chain pair joins a model a restricted key may reach with a target the chain names.

Response Cache Metrics¶

The eviction counter is exact for the in-memory LRU store. Redis and S3 perform expiry or eviction server-side and do not expose an eviction event to the router, so those backends do not fabricate a value.

Redis Cache Backend Metrics¶

These metrics are populated when the Redis cache backend is active (backend: redis).

KV Event Consumer Metrics¶

These metrics are populated when KV event consumers are active (src/infrastructure/kv_index/). vLLM, SGLang, and TensorRT-LLM listeners may use router salt echo to preserve the same prefix identity without detokenizing every block event. All backend label values are sanitized to prevent cardinality explosion.

Prefix Routing Metrics¶

These metrics track prefix-aware sticky routing decisions and backend distribution.

continuum_prefix_routing_requests_total is recorded once per backend selection and only when the PrefixAwareHash strategy arm ran, so it stays at zero under any other selection_strategy. The three labels map to the ring walk directly:

• prefix_hash: the prefix-hashed owner took the request, either because it was under the CHWBL load cap or because every backend was at the cap and the full-lap fallback kept the owner to preserve prefix locality.
• overflow: the owner was at or over the load cap and a later node clockwise on the ring took the request.
• fallback: the request carried no prefix key, so the model name was hashed and the ring was walked with no load cap. A steady 100% fallback rate almost always means prefix_routing.enabled is false.

unknown is a sanitizer output, not a routing path: it appears only if a label outside the allowlist ever reaches the recorder.

Key PromQL Queries¶

# Prefix routing hit rate (% of requests using prefix hash vs fallback)
sum(rate(continuum_prefix_routing_requests_total{strategy="prefix_hash"}[5m])) /
sum(rate(continuum_prefix_routing_requests_total[5m]))

# Overflow rate (CHWBL load balancing activations)
rate(continuum_prefix_routing_requests_total{strategy="overflow"}[5m])

# Backend load distribution (should be roughly even)
continuum_prefix_routing_backend_distribution

KV Cache Index Metrics¶

These metrics track the KV cache index subsystem including index state, query performance, routing decisions, and overlap scoring.

routing_engine_load_decisions_total is recorded once per scoring pass while routing.engine_load.enabled holds (plus once per refused selection when the admission hint trips):

• engine_load: The engine-load term actively ranked candidates and backend names the engine-preferred candidate.
• stale_fallback: No candidate had a fresh engine snapshot, so the pass matched base scoring (backend="none").
• hysteresis_hold: Fresh data existed, but the waiting_requests spread stayed inside the balance thresholds.
• admission_reject: The saturation admission hint refused selection because every healthy candidate reported kv_cache_usage above the threshold (backend="none").

routing_usage_weight_decisions_total is recorded once per scoring pass that the usage-weight term decides or cannot settle, while routing.usage_weight.enabled holds. A pass whose selection the KV overlap or engine-load terms already make keeps its KV metrics and is not counted here:

• applied: The usage-weight term strictly preferred one candidate and decided the pick, and backend names that candidate.
• no_declaration: No eligible candidate declared provider_tpm or provider_rpm, so the term had nothing to rank (backend="none").
• no_differentiation: At least one candidate declared a limit, but the term produced no strict preference (all participants exhausted at zero, equal remaining shares, or weight zero), so the configured strategy decided (backend="none").

The term reads the router-local consumption window, so there is no stale reason: the window either holds this window's usage or starts at zero.

routing_quota_aware_decisions_total is recorded once per candidate in a scoring pass while routing.quota_aware.enabled and routing.usage_weight.enabled both hold, and backend names that candidate:

• observed_used: A fresh upstream snapshot supplied at least one usable remaining share and replaced the router-local estimate for that axis.
• stale: A snapshot exists but is older than routing.quota_aware.ttl, so the declared estimate stayed in place.
• absent: No response carrying recognized quota headers has been seen for this candidate.
• unusable: A fresh snapshot exists, but none of its dimensions carried both a remaining value and a positive -limit denominator, so it could not rank.

Backend label values pass through the standard backend label sanitizer. The gate, load normalization, cached result, and backend label are scoped to the request's exact live candidate set, so an engine excluded by model visibility, health, retry state, or per-key permissions cannot influence or be named by that scoring decision.

Both routing labels are recorded at the same seam, once per selection: kv_aware when the KV overlap scorer won and its backend was forwarded to, and fallback when the scorer ran with a prefix key but no backend cleared the pool's compiled-in combined-score threshold of 0.3, so the configured selection_strategy decided instead. A selection the scorer never attempted (no scorer registered, or no prefix key) records neither, so the two counters sum to the number of scored selections rather than to total traffic. The histogram observes the winning combined score of a kv_aware decision, which is the full weighted sum of overlap, load, and health, not the overlap term on its own.

The tier label says where the chosen backend holds the request prefix according to the KV index, read from the same scoring pass that picked it: gpu_hot when the prefix blocks are resident on the accelerator, storage_warm when the index recorded them as offloaded to a storage tier (a KV connector offload event) so the engine has to reload them, and none for every fallback decision and for a kv_aware winner that holds no index entry for the prefix. A value outside that set is recorded as unknown, which no production path writes. Summing over tier gives the per-decision totals.

Key PromQL Queries¶

# KV-aware routing ratio
sum(rate(continuum_kv_index_routing_decisions_total{decision="kv_aware"}[5m])) /
sum(rate(continuum_kv_index_routing_decisions_total[5m]))

# Share of KV-aware decisions that picked a warm (offloaded) holder
sum(rate(continuum_kv_index_routing_decisions_total{decision="kv_aware",tier="storage_warm"}[5m])) /
sum(rate(continuum_kv_index_routing_decisions_total{decision="kv_aware"}[5m]))

# Average overlap score for routed requests
histogram_quantile(0.5, rate(continuum_kv_index_overlap_score_bucket[5m]))

# KV index query P99 latency
histogram_quantile(0.99, rate(continuum_kv_index_query_latency_seconds_bucket[5m]))

# Event source connection health
continuum_kv_index_event_source_status{status="connected"}

Smart Routing Metrics¶

These metrics cover the smart routing pipeline, including the LLM-based classifier.

Classification and Routing¶

Load Management¶

LLM Classifier¶

smart_routing_hybrid_gate_total records outcome="routed" when the rule classifier confidence reached rule.confidence_threshold, and outcome="escalated" when the gate escalated to the LLM classifier or System One (counted whether or not the downstream call was configured, load-shed, or cached, and regardless of what it returned).

System One Classifier¶

The decision calls themselves are counted by the shared client under systemone_requests_total{consumer="classifier", result} and timed by systemone_request_duration_seconds{consumer="classifier"} (see System One Decision Metrics). Results carry classifier_type="system_one" in the classification counter and duration histogram above.

Aggregate and Operational¶

Key PromQL Queries¶

# Smart routing request rate by policy
rate(smart_routing_requests_total[5m])

# Tier usage distribution
sum by(tier) (rate(smart_routing_tier_usage_total[5m]))

# LLM classifier cache hit rate
rate(smart_routing_llm_classifier_cache_hits_total[5m]) /
rate(smart_routing_llm_classifier_calls_total[5m])

# LLM classifier P95 latency
histogram_quantile(0.95, rate(smart_routing_llm_classifier_duration_seconds_bucket[5m]))

# LLM classifier fallback rate (reliability indicator)
rate(smart_routing_llm_classifier_fallbacks_total[5m]) /
rate(smart_routing_llm_classifier_calls_total[5m])

# Share of requests the hybrid gate escalates, per method
sum by(method) (rate(smart_routing_hybrid_gate_total{outcome="escalated"}[1h])) /
sum by(method) (rate(smart_routing_hybrid_gate_total[1h]))

# Fraction of requests classified by LLM vs rule-based
rate(smart_routing_classifications_total{classifier_type="llm_based"}[5m]) /
rate(smart_routing_classifications_total[5m])

# System One shadow agreement rate per field (run shadow mode before switching method)
sum by(field) (rate(smart_routing_shadow_agreement_total{agree="true"}[1h])) /
sum by(field) (rate(smart_routing_shadow_agreement_total[1h]))

# Share of System One answers discarded for low confidence (tune min_confidence)
rate(smart_routing_system_one_classifier_fallbacks_total{reason="low_confidence"}[5m]) /
rate(systemone_requests_total{consumer="classifier",result="ok"}[5m])

# Policy evaluation success rate
sum by(policy_name) (rate(smart_routing_policy_evaluations_total{result="matched"}[5m]))

Business Metrics¶

Model label formatting and cardinality rules:

• Verbatim model IDs: The model label on model_usage_total, model_tokens_processed, llm_tokens_total, and smart_routing_model_availability, as well as source_model and target_model on smart_routing_decisions_total and smart_routing_requests_total, use the same spelling rules as original_model and fallback_model on fallback metrics:
  • Case is preserved, and / and : are kept alongside Unicode letters, digits, -, _, and . (other characters become _, values truncate at 128 bytes on a character boundary, and an empty ID becomes unknown).
  • Names like meta-llama/Llama-3.1-8B-Instruct, llama3:8b, and hf.co/org/repo:Q4_K_M appear as written, and Qwen/Qwen3-8B and qwen/qwen3-8b remain distinct series.
  • Non-model labels (backend, api_key_id, type) continue to use the lowercasing sanitizer.
• Cardinality guard (metrics.max_model_labels): Bounds distinct models on model_usage_total, model_tokens_processed, and llm_tokens_total (1,000 by default) using the formatted spelling. Smart routing families are not gated by max_model_labels.
• source_model folding on smart_routing_requests_total: When smart_routing.intercept_all is enabled, every request is smart-routed regardless of the client-supplied model name. To keep cardinality bounded:
  • source_model is exported verbatim only when it matches smart_routing.virtual_model or a model the router currently serves.
  • Requests without model export none, and any other name (including model aliases) exports other (a served model literally named other or none shares that series).
  • The x-smart-route-source debug header (when smart_routing.debug_headers is enabled) and routing log lines still preserve the exact name sent by the client.

System One Decision Metrics¶

Recorded by the shared System One decision client (src/services/systemone/), which the guardrail provider, the smart-routing classifier, the action-safety evaluator, the answer-quality fallback check (consumer="fallback_quality"), and the usage quality judgment (consumer="usage_quality") use to reach a decision model. consumer is the calling seam, so a shared Jev key's rate limit can be attributed to whichever consumer spent it.

Action-Safety Metrics¶

Recorded by the System One evaluator of safeguards.action_safety in its default advisory mode. Both labels take a verdict (allow, deny, indeterminate), so the family has at most nine series. A series whose two labels differ counts disagreements: for example {would_verdict="allow", actual_verdict="deny"} is an action the decision model would have allowed that the chat-model evaluator denied. Read these pairs before switching to mode: enforce.

Guardrail Metrics¶

Exported when guardrails are configured and the metrics feature is enabled. Every guardrail decision is recorded so operators can observe what a policy does (or would do, in monitor mode) before and after enforcement.

Label values:

• stage is input, output, or streaming.
• result is allow, block, transform, or flag.
• kind is timeout or error.
• mode is monitor or enforce. Because guardrail_verdicts_total carries mode, monitor-mode verdicts are visible even though they never gate a request, which is what makes the monitor-then-enforce rollout observable.
• route is the Hub tier guardrail_route name in effect, or "none" when no tier route was attached. At most 16 distinct route label values are tracked; overflow collapses to "other" to bound cardinality.
• provider is a configured provider name, or one of the two reserved match-list labels: match_list_deny when a guardrails.deny rule decided the check, match_list_allow when an allow rule did. A deny-list block also carries category="deny_list" on guardrail_blocks_total. See Allow and deny lists.
• strategy is the strategy in effect when the cap tripped (buffer_full / chunked / monitor); outcome is how output checking continued past it (degraded_chunked / compacted / truncated). See the buffer cap section for what each outcome means for the safety guarantee. strategy="monitor" is not a safety event: it means a monitor-mode stream was long enough that only its first 4 MiB was observed.

Key PromQL Queries¶

# What would be blocked, broken down by category (monitor-mode tuning)
sum by (category) (rate(guardrail_blocks_total[1h]))

# Block rate per stage after enforcement
sum by (stage) (rate(guardrail_verdicts_total{result="block", mode="enforce"}[5m]))

# Provider error rate (timeouts vs hard failures)
sum by (provider, kind) (rate(guardrail_errors_total[5m]))

# P95 guardrail check latency per provider
histogram_quantile(0.95, sum by (le, provider) (rate(guardrail_check_duration_seconds_bucket[5m])))

For the full guardrail guide (concepts, providers, configuration, and the threshold-tuning workflow), see Guardrails.

Per-API-Key LLM Token Usage¶

The router publishes a per-API-key breakdown of LLM token consumption so operators can answer questions like "which key consumed the most completion tokens last hour?" or "how many prompt tokens did team X spend on model Y today?". This data is independent of the model_tokens_processed aggregate counter and supports capacity planning, fair-use enforcement, and external cost attribution.

Metric Definition¶

kind is one of:

• prompt: tokens in the upstream request prompt
• completion: tokens in the upstream response completion

Label and series rules for usage counters:

• outcome: Indicates how the attempt ended, from the closed set completed, length, client_cancelled, timeout_first_byte, timeout_mid_stream, upstream_error, fallback_source, guardrail_blocked, usage_missing, router_restart (a normal completion exports completed; a token-limited completion exports length; summing across all labels is unchanged).
• usage_records_total: Uses the same outcome set, token_source (engine_final, engine_continuous, router_delta_count, engine_tokenize, char_estimate, none), and purpose (client, guardrail, classifier, action_safety, quality_judgment). All values come from closed router-side enums (unknown is never exported), and the family stays at zero on builds without the control-plane feature. See "Usage record outcome and attempt identity" in the architecture document for details.
• usage_missing_total{backend}: Counts streams that reached their relay's normal end without an engine usage block, so the usage record uses the router's delta count (token_source="router_delta_count", or none if nothing streamed) with outcome="usage_missing":
  • Streams ending with an upstream error event (outcome="upstream_error") or interrupted mid-stream (client_cancelled, timeout_first_byte, timeout_mid_stream, guardrail_blocked, fallback_source) across OpenAI-wire, Unix-socket, chat pipeline, mid-stream fallback, Anthropic Messages, /v1/responses, responses_only bridge, and transformer relays record their respective outcome and are not counted here (whereas a /v1/responses Chat Completions conversion that completes without engine usage is counted).
  • Incremented by finalized stream sinks before queuing the usage event on every build with the metrics feature; if a full usage channel drops the event afterward, compare against usage_records_total{outcome="usage_missing"} for records actually formed.
  • A rising rate on one backend indicates its engine is not sending stream_options.include_usage usage, a proxy is stripping it, or the terminal SSE frame is being lost. See "Router-observed output counting and the stream-usage finalizer" in the architecture document.

usage_export_records_evicted_total counts the records the per-attempt ring behind GET /admin/usage/export dropped: retention for a record older than admin.stats.usage_export.retention (on the next append or on the hourly series prune), max_records for one evicted to stay within admin.stats.usage_export.max_records. It stays at zero while the export is disabled.

Both OpenAI-compatible (prompt_tokens / completion_tokens) and Anthropic (input_tokens / output_tokens) response shapes are normalized into the same counter. The router also injects stream_options.include_usage=true on OpenAI-compat streaming requests so usage data arrives in the final SSE chunk regardless of client behavior.

api_key_id Derivation¶

api_key_id is never the raw API key. The router derives a stable, non-reversible identifier in this priority order:

1. If the request's bearer token matches a configured API-key entry, the entry's id field is used (e.g., key-production-1).
2. Otherwise, the router computes SHA-256 over the raw token and uses the first 12 hex characters prefixed with k_ (e.g., k_3f5a7c9b1e2d).
3. If no token is presented, the literal value anonymous is used.

All label values flow through the existing CardinalityManager so a runaway/rotating-key attack cannot exhaust Prometheus series.

Annotation Labels and api_key_info¶

Each configured API key may carry an optional free-form annotations: { key: value } map. Operators declare which annotation keys become Prometheus labels via the global metrics.annotation_labels allowlist; everything else stays internal.

Configuration schema (under the existing api_keys block):

api_keys:
  api_keys:
      - key: "${API_KEY_1}"
        id: "key-production-1"
        user_id: "user-admin"
        organization_id: "org-main"
        annotations:
            email: "ops@example.com"
            team: "platform"
            environment: "prod"
            owner: "alice"

metrics:
    enabled: true
    annotation_labels: [email, team]            # Allowlist of label keys

Reserved annotation keys (recommended canonical names, not enforced): email, uuid, owner, team, environment. Operators may add custom keys.

When metrics.annotation_labels is non-empty, the router publishes api_key_info{api_key_id, email, team, ...} = 1 once per known key. Use PromQL joins to project the metadata onto llm_tokens_total without bloating its label set:

# Tokens per email (sums prompt + completion, last 24h)
sum by (email) (
  increase(llm_tokens_total[24h])
  * on (api_key_id) group_left(email) api_key_info
)

Cardinality and Hot-Reload¶

• api_key_id cardinality is bounded at 1000 by default.
• Hot-reload of API-key annotations is supported via the existing config-reload pipeline. The api_key_info info-metric is republished atomically on every reload; counter values for llm_tokens_total are never reset.
• The label set on api_key_info (i.e., the contents of annotation_labels) is frozen at startup. Adding or removing keys from the allowlist requires a restart, since Prometheus does not allow renaming labels on a registered metric.

Example PromQL Queries¶

# Total prompt tokens consumed per API key in the last hour
sum by (api_key_id) (
  increase(llm_tokens_total{kind="prompt"}[1h])
)

# Top 10 keys by completion tokens in the last 24h
topk(10,
  sum by (api_key_id) (
    increase(llm_tokens_total{kind="completion"}[24h])
  )
)

# Tokens grouped by team (requires team in annotation_labels)
sum by (team) (
  increase(llm_tokens_total[24h])
  * on (api_key_id) group_left(team) api_key_info
)

# Combined prompt+completion rate per model (tokens/sec)
sum by (model) (rate(llm_tokens_total[5m]))

# Per-key consumption by backend (useful for cost attribution)
sum by (api_key_id, backend) (
  increase(llm_tokens_total[24h])
)

Grafana Panel Example¶

A simple Grafana stat panel showing the top 10 teams by completion tokens over the last 24 hours:

{
    "title": "Top 10 teams by completion tokens (24h)",
    "type": "stat",
    "targets": [
        {
            "expr": "topk(10, sum by (team) (increase(llm_tokens_total{kind=\"completion\"}[24h]) * on (api_key_id) group_left(team) api_key_info))",
            "legendFormat": "{{team}}"
        }
    ],
    "options": {
        "reduceOptions": {
            "values": false,
            "calcs": ["lastNotNull"]
        }
    }
}

For tracking spend trends, pair this with a time-series panel using rate(llm_tokens_total[5m]) grouped by team or model.

Verification Steps¶

After enabling the feature:

1. Issue a chat-completion request with a configured API key.
2. Scrape /metrics and confirm llm_tokens_total{...} and api_key_info{...} series appear.
3. For streaming, verify the counter still increments: usage is captured from the final SSE chunk. The router injects stream_options.include_usage=true automatically for OpenAI-compat backends so this works regardless of client behavior.
4. Inspect /metrics cardinality on a typical workload (e.g., wc -l < /metrics) to confirm no regression versus the prior baseline.

Usage Ledger Durability¶

Four families describe what happens to control-plane usage records between the producer and a hub acknowledgment. They exist on every build with the metrics feature; without control_plane.enabled they stay at zero.

reason is one of a closed set, every value present at zero from startup:

• channel_full: the 10,000-slot producer channel was full or closed.
• buffer_overflow: the memory buffer overflowed with the spool disabled (control_plane.usage_spool_max_bytes: 0) or unusable, so the oldest record was dropped.
• spool_overflow: the spool byte cap deleted the oldest file, refused a batch larger than the cap, or a spool write failed. The records' router_usage_seq values are retired.
• batch_rejected_413: the hub refused a single-record batch as too large. A larger batch the hub refuses with 413 is re-split and retried at half the size, so its records are not counted here.
• ack_oversized: no longer incremented. A 2xx acknowledgment over the decode bound means the hub processed the batch, so the batch counts as delivered and the router only logs a warning; the label stays in the set at zero for query compatibility.
• wal_open_bound: the WAL writer already tracks its maximum of 65,536 open attempts, so a new attempt's start line was not written. The record itself is still pushed when the attempt ends, but a crash before then cannot replay it as router_restart. A rate-limited warning names the bound.
• wal_dropped: a write-ahead log line was not written (the channel stayed full for the 250 ms send wait, or the log was already saturated by such a wait; the boot replay's open-attempt bound; or a write failure). A dropped end line leaves a false open start that a later crash would replay as router_restart.

usage_durability_corrupt_total and operational signals:

• kind="wal_line": An unparseable line in <state_file>.usage-wal.jsonl was skipped. A WAL file that is a symlink, not a regular file, or not owner-only (0600) logs a warning and replays nothing; replayed attempts are counted under usage_records_total{outcome="router_restart"}.
• kind="spool_file": An unparseable, non-regular, symlink, or non-0600 spool batch file was deleted, or a spool batch written under a different router enrollment was deleted (its records also increment usage_records_dropped_total{reason="spool_overflow"}).
• Health indicators: A non-zero usage_spool_records that does not drain means the Hub is still refusing pushes; rate(usage_records_dropped_total{reason!~"wal_.*"}[5m]) > 0 indicates that the Hub's ledger is incomplete. The wal_open_bound and wal_dropped reasons lose crash-recovery coverage, not delivered records; alert on them separately.

Usage Ledger Tamper Evidence¶

Four families describe the hash chain over pushed usage batches and the local ledger export; see Tamper evidence for the hash and the gap rule. They exist on every build with the metrics feature; without control_plane.enabled they stay at zero.

Both target values are initialized to 0 at startup:

• usage_chain_gaps_total: Increments when a built batch is dropped due to HTTP 413, spool refusal, failed write, cap eviction, corrupt file, failed shutdown flush, unacknowledged batch at shutdown with the spool disabled, or a batch spooled under another enrollment. A rising count means the Hub's chain has router-declared holes.
• usage_ledger_export_failures_total{target="export"}: Increments on open or write failures (including paths refused as control_plane.state_file or its sidecars), a full writer channel, or a line refused at max_bytes with rotate: false. A value above 0 means the local export file has holes that the verifier will report as breaks.
• usage_ledger_export_failures_total{target="chain_state"}: Increments when writing <state_file>.usage-chain.json fails or a batch cannot be hashed. A value above 0 means the next restart may begin a fresh chain.

Integration¶

Prometheus Configuration¶

Complete Prometheus configuration example:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'continuum-router'
    static_configs:
      - targets: ['router1:9090', 'router2:9090']
    authorization:
      credentials: '<metrics-token>'
    metric_relabel_configs:
      # Drop high-cardinality metrics if needed
      - source_labels: [__name__]
        regex: 'http_request_duration_seconds_bucket'
        action: drop

Kubernetes Integration¶

For Kubernetes deployments, use ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: continuum-router
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: continuum-router
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics
      bearerTokenSecret:
        name: continuum-router-metrics
        key: token

The Helm chart can render this object with serviceMonitor.enabled=true when the Prometheus Operator CRD is installed. For clusters without the operator, the repository also ships a standalone Prometheus Kustomize bundle with Kubernetes endpoint discovery and the checked-in alert rules:

kubectl apply -k monitoring/prometheus
kubectl -n monitoring rollout status deployment/continuum-router-prometheus
kubectl -n monitoring port-forward service/continuum-router-prometheus 9090:9090

The standalone profile requests a 10 GiB ReadWriteOnce PersistentVolumeClaim with 15-day retention. Confirm the default StorageClass and resize the claim for expected series volume, or use a managed Prometheus service for highly available monitoring. Review its cluster-wide discovery RBAC and NetworkPolicy before applying it.

Grafana Dashboard¶

monitoring/grafana/dashboards/router-overview.json contains these panels:

• Request rate
• Error rate
• P95 latency
• Backend health
• Request rate by endpoint
• Response-time percentiles
• Model usage distribution
• Current in-flight load across backends
• Engine requests (running / waiting), from engine_stats
• Engine KV cache usage, from engine_stats
• Engine-load routing decisions by reason, from routing.engine_load

To import it:

1. Open Grafana.
2. Select Dashboards → Import.
3. Upload monitoring/grafana/dashboards/router-overview.json.
4. Select the Prometheus data source and import.

Alerting¶

Pre-configured alert rules are available in monitoring/prometheus/alerts.yml:

Critical Alerts¶

{% raw %}

- alert: HighErrorRate
  expr: sum(rate(http_requests_total{status_code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
  for: 5m
  annotations:
    summary: "High error rate: {{ $value | humanizePercentage }}"

Warning Alerts¶

{% raw %}

- alert: HighLatency
  expr: histogram_quantile(0.95, http_request_duration_seconds) > 1
  for: 5m
  annotations:
    summary: "P95 latency above 1s: {{ $value | humanizeDuration }}"

- alert: TimeoutErrors
  expr: sum(rate(timeout_errors_total[5m])) > 0.1
  for: 10m
  annotations:
    summary: "Frequent timeout errors: {{ $value }} per second"

Examples¶

Query Examples¶

Request Rate by Status¶

sum(rate(http_requests_total[5m])) by (status)

P95 Latency by Endpoint¶

histogram_quantile(0.95, 
  sum(rate(http_request_duration_seconds_bucket[5m])) by (endpoint, le)
)

Backend Load Overview¶

backend_current_load

Model Usage Ranking¶

topk(10, sum(rate(model_usage_total[1h])) by (model))

Error Rate Percentage¶

sum(rate(http_requests_total{status_code=~"5.."}[5m])) /
sum(rate(http_requests_total[5m])) * 100

Programmatic Access¶

You can also access metrics programmatically:

import os
import requests
from prometheus_client.parser import text_string_to_metric_families

# Fetch metrics
response = requests.get(
    'http://localhost:9090/metrics',
    headers={'Authorization': f"Bearer {os.environ['METRICS_TOKEN']}"},
)
metrics = text_string_to_metric_families(response.text)

# Process metrics
for family in metrics:
    for sample in family.samples:
        if sample.name == 'http_requests_total':
            print(f"Endpoint: {sample.labels['endpoint']}, Count: {sample.value}")

Custom Metrics Collection¶

#!/bin/bash
# Collect metrics every 30 seconds and save to file

while true; do
  timestamp=$(date +%s)
  curl -s -H "Authorization: Bearer $METRICS_TOKEN" \
    http://localhost:9090/metrics > "metrics_${timestamp}.txt"
  sleep 30
done

Best Practices¶

1. Label Cardinality¶

Keep label cardinality low to prevent metric explosion:

# Good: Low cardinality
labels:
  status: "200"  # ~5 possible values
  method: "GET"  # ~7 possible values

---

# Bad: High cardinality
labels:
  user_id: "12345"  # Unbounded
  request_id: "abc-123"  # Unique per request

2. Metric Naming¶

Follow Prometheus naming conventions:

• Use snake_case
• Include units in metric names (_seconds, _bytes, _total)
• Use standard prefixes (http_, backend_, model_)

3. Dashboard Design¶

• Group related metrics together
• Use appropriate visualization types (gauge for current values, graph for time series)
• Include both absolute values and rates
• Set reasonable refresh intervals (15-30s for real-time, 1-5m for historical)

4. Alert Configuration¶

• Use appropriate evaluation periods (for: 5m to avoid flapping)
• Include context in alert descriptions
• Set up alert routing based on severity
• Test alerts in staging before production

5. Performance Considerations¶

• Disable optional metrics if not needed
• Use recording rules for complex queries
• Implement proper metric retention policies
• Consider using remote storage for long-term retention

6. Security¶

• Protect metrics endpoint if sensitive data is exposed
• Use TLS for Prometheus scraping in production
• Implement authentication for Grafana dashboards
• Audit metric access logs

Troubleshooting¶

Metrics Not Appearing¶

1. Check if metrics are enabled in configuration
2. Verify the metrics endpoint is accessible
3. Check Prometheus target status
4. Review router logs for metric initialization errors

High Memory Usage¶

1. Review cardinality limits
2. Check for unbounded labels
3. Reduce histogram buckets if needed
4. Enable metric expiration

Incorrect Values¶

1. Verify metric types (counter vs gauge)
2. Check aggregation functions
3. Review label selectors
4. Validate time ranges

Additional Resources¶

• Prometheus Documentation
• Grafana Documentation
• Prometheus Best Practices
• Monitoring Distributed Systems