Metrics and Monitoring¶

This document describes the metrics and monitoring capabilities of the Continuum Router.

Table of Contents¶

• Overview
• Quick Start
• Configuration
• Available Metrics
• Integration
• Grafana Dashboard
• Alerting
• Examples
• Best Practices

Overview¶

The Continuum Router exposes Prometheus-compatible metrics for monitoring system health, performance, and usage patterns. The metrics system is designed to be:

• Lightweight: Minimal performance overhead
• Broad coverage: Covers HTTP, backend, routing, model, streaming, and cache subsystems
• Production-ready: Includes cardinality limits and proper labeling
• Easy to integrate: Works with standard Prometheus/Grafana setups

For restart-survival history without standing up a full Prometheus stack, see the Persistent Metrics Log. It snapshots the registry to a local SQLite store and exposes recent history via GET /admin/metrics/history.

Quick Start¶

1. Enable Metrics¶

Metrics are enabled by default. The metrics endpoint is available at /metrics:

# View metrics
curl http://localhost:9090/metrics

2. Configure Prometheus¶

Add the router as a target in your prometheus.yml:

scrape_configs:
  - job_name: 'continuum-router'
    static_configs:
      - targets: ['localhost:9090']
    scrape_interval: 15s

3. Import Grafana Dashboard¶

Import the provided dashboard from monitoring/grafana/dashboards/router-overview.json.

Configuration¶

Metrics configuration is done through the main config file:

metrics:
  enabled: true
  port: 9090
  path: /metrics
  max_model_labels: 1000
  max_backend_labels: 100
  cardinality_limits:
    max_models: 1000
    max_endpoints: 100
    max_error_types: 100
    enable_sampling: false
    sampling_rate: 1.0
  # External Prometheus is the durable store for this profile.
  persistence:
    enabled: false

Environment Variables¶

You can also configure metrics using environment variables:

# Enable/disable metrics
METRICS_ENABLED=true

# Change metrics endpoint
METRICS_ENDPOINT=/custom/metrics

# Enable optional metrics
METRICS_ENABLE_BODY_SIZE=true

Available Metrics¶

HTTP Metrics¶

The HTTP middleware reports backend="unknown" because backend selection happens inside handlers and not every route selects a backend. Per-backend traffic and load are available through the backend and prefix-routing metric families; client-supplied headers are never trusted for metric attribution.

Error and Retry Metrics¶

Circuit Breaker Metrics¶

Populated by ordinary proxy traffic when the metrics feature and a circuit_breaker are both enabled.

Request Parameter Policy Metrics¶

Both families use closed, bounded labels. source is one of none, local, hub, or intersection; neither family exports request values, model ids, tier ids, key ids, or policy cursors.

Backend Metrics¶

Model Service Metrics¶

Routing telemetry is exported by the live KV cache and smart routing families below. Unsupported members of the former generic routing and model-service collectors are not registered.

Active streaming response bodies contribute to http_active_connections. The router does not currently export generic streaming duration or post-response error metrics; mid-stream fallback has the dedicated live metrics below.

Mid-Stream Fallback Metrics¶

These metrics are emitted when the mid-stream fallback feature is enabled (streaming.mid_stream_fallback.enabled: true).

Reason Label Values for streaming_fallback_total¶

Key PromQL Queries¶

# Mid-stream fallback rate
rate(streaming_fallback_total[5m])

# Fallback recovery success rate
sum(rate(streaming_fallback_success_total[5m])) /
sum(rate(streaming_fallback_total[5m]))

# Median accumulated tokens at fallback trigger
histogram_quantile(0.5, rate(streaming_fallback_accumulated_tokens_bucket[5m]))

Fallback Metrics¶

Response Cache Metrics¶

The eviction counter is exact for the in-memory LRU store. Redis and S3 perform expiry or eviction server-side and do not expose an eviction event to the router, so those backends do not fabricate a value.

Redis Cache Backend Metrics¶

These metrics are populated when the Redis cache backend is active (backend: redis).

KV Event Consumer Metrics¶

These metrics are populated when vLLM KV event consumers are active (src/infrastructure/kv_index/). All backend label values are sanitized to prevent cardinality explosion.

Prefix Routing Metrics¶

These metrics track prefix-aware sticky routing decisions and backend distribution.

Key PromQL Queries¶

# Prefix routing hit rate (% of requests using prefix hash vs fallback)
sum(rate(continuum_prefix_routing_requests_total{strategy="prefix_hash"}[5m])) /
sum(rate(continuum_prefix_routing_requests_total[5m]))

# Overflow rate (CHWBL load balancing activations)
rate(continuum_prefix_routing_requests_total{strategy="overflow"}[5m])

# Backend load distribution (should be roughly even)
continuum_prefix_routing_backend_distribution

KV Cache Index Metrics¶

These metrics track the KV cache index subsystem including index state, query performance, routing decisions, and overlap scoring.

Key PromQL Queries¶

# KV-aware routing ratio
sum(rate(continuum_kv_index_routing_decisions_total{decision="kv_aware"}[5m])) /
sum(rate(continuum_kv_index_routing_decisions_total[5m]))

# Average overlap score for routed requests
histogram_quantile(0.5, rate(continuum_kv_index_overlap_score_bucket[5m]))

# KV index query P99 latency
histogram_quantile(0.99, rate(continuum_kv_index_query_latency_seconds_bucket[5m]))

# Event source connection health
continuum_kv_index_event_source_status{status="connected"}

Smart Routing Metrics¶

These metrics cover the smart routing pipeline, including the LLM-based classifier.

Classification and Routing¶

Load Management¶

LLM Classifier¶

Aggregate and Operational¶

Key PromQL Queries¶

# Smart routing request rate by policy
rate(smart_routing_requests_total[5m])

# Tier usage distribution
sum by(tier) (rate(smart_routing_tier_usage_total[5m]))

# LLM classifier cache hit rate
rate(smart_routing_llm_classifier_cache_hits_total[5m]) /
rate(smart_routing_llm_classifier_calls_total[5m])

# LLM classifier P95 latency
histogram_quantile(0.95, rate(smart_routing_llm_classifier_duration_seconds_bucket[5m]))

# LLM classifier fallback rate (reliability indicator)
rate(smart_routing_llm_classifier_fallbacks_total[5m]) /
rate(smart_routing_llm_classifier_calls_total[5m])

# Fraction of requests classified by LLM vs rule-based
rate(smart_routing_classifications_total{classifier_type="llm_based"}[5m]) /
rate(smart_routing_classifications_total[5m])

# Policy evaluation success rate
sum by(policy_name) (rate(smart_routing_policy_evaluations_total{result="matched"}[5m]))

Business Metrics¶

Guardrail Metrics¶

Exported when guardrails are configured and the metrics feature is enabled. Every guardrail decision is recorded so operators can observe what a policy does (or would do, in monitor mode) before and after enforcement.

Label values:

• stage is input, output, or streaming.
• result is allow, block, transform, or flag.
• kind is timeout or error.
• mode is monitor or enforce. Because guardrail_verdicts_total carries mode, monitor-mode verdicts are visible even though they never gate a request, which is what makes the monitor-then-enforce rollout observable.
• provider is a configured provider name, or one of the two reserved match-list labels: match_list_deny when a guardrails.deny rule decided the check, match_list_allow when an allow rule did. A deny-list block also carries category="deny_list" on guardrail_blocks_total. See Allow and deny lists.
• strategy is the strategy in effect when the cap tripped (buffer_full / chunked / monitor); outcome is how output checking continued past it (degraded_chunked / compacted / truncated). See the buffer cap section for what each outcome means for the safety guarantee. strategy="monitor" is not a safety event: it means a monitor-mode stream was long enough that only its first 4 MiB was observed.

Key PromQL Queries¶

# What would be blocked, broken down by category (monitor-mode tuning)
sum by (category) (rate(guardrail_blocks_total[1h]))

# Block rate per stage after enforcement
sum by (stage) (rate(guardrail_verdicts_total{result="block", mode="enforce"}[5m]))

# Provider error rate (timeouts vs hard failures)
sum by (provider, kind) (rate(guardrail_errors_total[5m]))

# P95 guardrail check latency per provider
histogram_quantile(0.95, sum by (le, provider) (rate(guardrail_check_duration_seconds_bucket[5m])))

For the full guardrail guide (concepts, providers, configuration, and the threshold-tuning workflow), see Guardrails.

Per-API-Key LLM Token Usage¶

The router publishes a per-API-key breakdown of LLM token consumption so operators can answer questions like "which key consumed the most completion tokens last hour?" or "how many prompt tokens did team X spend on model Y today?". This data is independent of the model_tokens_processed aggregate counter and supports capacity planning, fair-use enforcement, and external cost attribution.

Metric Definition¶

kind is one of:

• prompt — tokens in the upstream request prompt
• completion — tokens in the upstream response completion

Both OpenAI-compatible (prompt_tokens / completion_tokens) and Anthropic (input_tokens / output_tokens) response shapes are normalized into the same counter. The router also injects stream_options.include_usage=true on OpenAI-compat streaming requests so usage data arrives in the final SSE chunk regardless of client behavior.

api_key_id Derivation¶

api_key_id is never the raw API key. The router derives a stable, non-reversible identifier in this priority order:

1. If the request's bearer token matches a configured API-key entry, the entry's id field is used (e.g., key-production-1).
2. Otherwise, the router computes SHA-256 over the raw token and uses the first 12 hex characters prefixed with k_ (e.g., k_3f5a7c9b1e2d).
3. If no token is presented, the literal value anonymous is used.

All label values flow through the existing CardinalityManager so a runaway/rotating-key attack cannot exhaust Prometheus series.

Annotation Labels and api_key_info¶

Each configured API key may carry an optional free-form annotations: { key: value } map. Operators declare which annotation keys become Prometheus labels via the global metrics.annotation_labels allowlist; everything else stays internal.

Configuration schema (under the existing api_keys block):

api_keys:
  api_keys:
      - key: "${API_KEY_1}"
        id: "key-production-1"
        user_id: "user-admin"
        organization_id: "org-main"
        annotations:
            email: "ops@example.com"
            team: "platform"
            environment: "prod"
            owner: "alice"

metrics:
    enabled: true
    annotation_labels: [email, team]            # Allowlist of label keys

Reserved annotation keys (recommended canonical names, not enforced): email, uuid, owner, team, environment. Operators may add custom keys.

When metrics.annotation_labels is non-empty, the router publishes api_key_info{api_key_id, email, team, ...} = 1 once per known key. Use PromQL joins to project the metadata onto llm_tokens_total without bloating its label set:

# Tokens per email (sums prompt + completion, last 24h)
sum by (email) (
  increase(llm_tokens_total[24h])
  * on (api_key_id) group_left(email) api_key_info
)

Cardinality and Hot-Reload¶

• api_key_id cardinality is bounded at 1000 by default.
• Hot-reload of API-key annotations is supported via the existing config-reload pipeline. The api_key_info info-metric is republished atomically on every reload; counter values for llm_tokens_total are never reset.
• The label set on api_key_info (i.e., the contents of annotation_labels) is frozen at startup. Adding or removing keys from the allowlist requires a restart — Prometheus does not allow renaming labels on a registered metric.

Example PromQL Queries¶

# Total prompt tokens consumed per API key in the last hour
sum by (api_key_id) (
  increase(llm_tokens_total{kind="prompt"}[1h])
)

# Top 10 keys by completion tokens in the last 24h
topk(10,
  sum by (api_key_id) (
    increase(llm_tokens_total{kind="completion"}[24h])
  )
)

# Tokens grouped by team (requires team in annotation_labels)
sum by (team) (
  increase(llm_tokens_total[24h])
  * on (api_key_id) group_left(team) api_key_info
)

# Combined prompt+completion rate per model (tokens/sec)
sum by (model) (rate(llm_tokens_total[5m]))

# Per-key consumption by backend (useful for cost attribution)
sum by (api_key_id, backend) (
  increase(llm_tokens_total[24h])
)

Grafana Panel Example¶

A simple Grafana stat panel showing the top 10 teams by completion tokens over the last 24 hours:

{
    "title": "Top 10 teams by completion tokens (24h)",
    "type": "stat",
    "targets": [
        {
            "expr": "topk(10, sum by (team) (increase(llm_tokens_total{kind=\"completion\"}[24h]) * on (api_key_id) group_left(team) api_key_info))",
            "legendFormat": "{{team}}"
        }
    ],
    "options": {
        "reduceOptions": {
            "values": false,
            "calcs": ["lastNotNull"]
        }
    }
}

For tracking spend trends, pair this with a time-series panel using rate(llm_tokens_total[5m]) grouped by team or model.

Verification Steps¶

After enabling the feature:

1. Issue a chat-completion request with a configured API key.
2. Scrape /metrics and confirm llm_tokens_total{...} and api_key_info{...} series appear.
3. For streaming, verify the counter still increments — usage is captured from the final SSE chunk. The router injects stream_options.include_usage=true automatically for OpenAI-compat backends so this works regardless of client behavior.
4. Inspect /metrics cardinality on a typical workload (e.g., wc -l < /metrics) to confirm no regression versus the prior baseline.

Integration¶

Prometheus Configuration¶

Complete Prometheus configuration example:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'continuum-router'
    static_configs:
      - targets: ['router1:9090', 'router2:9090']
    metric_relabel_configs:
      # Drop high-cardinality metrics if needed
      - source_labels: [__name__]
        regex: 'http_request_duration_seconds_bucket'
        action: drop

Kubernetes Integration¶

For Kubernetes deployments, use ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: continuum-router
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: continuum-router
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics

The Helm chart can render this object with serviceMonitor.enabled=true when the Prometheus Operator CRD is installed. For clusters without the operator, the repository also ships a standalone Prometheus Kustomize bundle with Kubernetes endpoint discovery and the checked-in alert rules:

kubectl apply -k monitoring/prometheus
kubectl -n monitoring rollout status deployment/continuum-router-prometheus
kubectl -n monitoring port-forward service/continuum-router-prometheus 9090:9090

The standalone profile requests a 10 GiB ReadWriteOnce PersistentVolumeClaim with 15-day retention. Confirm the default StorageClass and resize the claim for expected series volume, or use a managed Prometheus service for highly available monitoring. Review its cluster-wide discovery RBAC and NetworkPolicy before applying it.

Grafana Dashboard¶

monitoring/grafana/dashboards/router-overview.json contains these panels:

• Request rate
• Error rate
• P95 latency
• Backend health
• Request rate by endpoint
• Response-time percentiles
• Model usage distribution
• Current in-flight load across backends

To import it:

1. Open Grafana.
2. Select Dashboards → Import.
3. Upload monitoring/grafana/dashboards/router-overview.json.
4. Select the Prometheus data source and import.

Alerting¶

Pre-configured alert rules are available in monitoring/prometheus/alerts.yml:

Critical Alerts¶

{% raw %}

- alert: HighErrorRate
  expr: sum(rate(http_requests_total{status_code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
  for: 5m
  annotations:
    summary: "High error rate: {{ $value | humanizePercentage }}"

Warning Alerts¶

{% raw %}

- alert: HighLatency
  expr: histogram_quantile(0.95, http_request_duration_seconds) > 1
  for: 5m
  annotations:
    summary: "P95 latency above 1s: {{ $value | humanizeDuration }}"

- alert: TimeoutErrors
  expr: sum(rate(timeout_errors_total[5m])) > 0.1
  for: 10m
  annotations:
    summary: "Frequent timeout errors: {{ $value }} per second"

Examples¶

Query Examples¶

Request Rate by Status¶

sum(rate(http_requests_total[5m])) by (status)

P95 Latency by Endpoint¶

histogram_quantile(0.95, 
  sum(rate(http_request_duration_seconds_bucket[5m])) by (endpoint, le)
)

Backend Load Overview¶

backend_current_load

Model Usage Ranking¶

topk(10, sum(rate(model_usage_total[1h])) by (model))

Error Rate Percentage¶

sum(rate(http_requests_total{status_code=~"5.."}[5m])) /
sum(rate(http_requests_total[5m])) * 100

Programmatic Access¶

You can also access metrics programmatically:

import requests
from prometheus_client.parser import text_string_to_metric_families

# Fetch metrics
response = requests.get('http://localhost:9090/metrics')
metrics = text_string_to_metric_families(response.text)

# Process metrics
for family in metrics:
    for sample in family.samples:
        if sample.name == 'http_requests_total':
            print(f"Endpoint: {sample.labels['endpoint']}, Count: {sample.value}")

Custom Metrics Collection¶

#!/bin/bash
# Collect metrics every 30 seconds and save to file

while true; do
  timestamp=$(date +%s)
  curl -s http://localhost:9090/metrics > "metrics_${timestamp}.txt"
  sleep 30
done

Best Practices¶

1. Label Cardinality¶

Keep label cardinality low to prevent metric explosion:

# Good: Low cardinality
labels:
  status: "200"  # ~5 possible values
  method: "GET"  # ~7 possible values

---

# Bad: High cardinality
labels:
  user_id: "12345"  # Unbounded
  request_id: "abc-123"  # Unique per request

2. Metric Naming¶

Follow Prometheus naming conventions:

• Use snake_case
• Include units in metric names (_seconds, _bytes, _total)
• Use standard prefixes (http_, backend_, model_)

3. Dashboard Design¶

• Group related metrics together
• Use appropriate visualization types (gauge for current values, graph for time series)
• Include both absolute values and rates
• Set reasonable refresh intervals (15-30s for real-time, 1-5m for historical)

4. Alert Configuration¶

• Use appropriate evaluation periods (for: 5m to avoid flapping)
• Include context in alert descriptions
• Set up alert routing based on severity
• Test alerts in staging before production

5. Performance Considerations¶

• Disable optional metrics if not needed
• Use recording rules for complex queries
• Implement proper metric retention policies
• Consider using remote storage for long-term retention

6. Security¶

• Protect metrics endpoint if sensitive data is exposed
• Use TLS for Prometheus scraping in production
• Implement authentication for Grafana dashboards
• Audit metric access logs

Troubleshooting¶

Metrics Not Appearing¶

1. Check if metrics are enabled in configuration
2. Verify the metrics endpoint is accessible
3. Check Prometheus target status
4. Review router logs for metric initialization errors

High Memory Usage¶

1. Review cardinality limits
2. Check for unbounded labels
3. Reduce histogram buckets if needed
4. Enable metric expiration

Incorrect Values¶

1. Verify metric types (counter vs gauge)
2. Check aggregation functions
3. Review label selectors
4. Validate time ranges

Additional Resources¶

• Prometheus Documentation
• Grafana Documentation
• Prometheus Best Practices
• Monitoring Distributed Systems