Metrics and Monitoring¶

This document describes the metrics and monitoring capabilities of the Continuum Router.

Table of Contents¶

• Overview
• Quick Start
• Configuration
• Available Metrics
• Integration
• Grafana Dashboard
• Alerting
• Examples
• Best Practices

Overview¶

The Continuum Router exposes Prometheus-compatible metrics for monitoring system health, performance, and usage patterns. The metrics system is designed to be:

• Lightweight: Minimal performance overhead
• Broad coverage: Covers HTTP, backend, routing, model, streaming, and cache subsystems
• Production-ready: Includes cardinality limits and proper labeling
• Easy to integrate: Works with standard Prometheus/Grafana setups

For restart-survival history without standing up a full Prometheus stack, see the Persistent Metrics Log. It snapshots the registry to a local SQLite store and exposes recent history via GET /admin/metrics/history.

Metrics answer how much and how often. For per-request causality, which backend was chosen, how many attempts a request took, and why each attempt ended the way it did, see Distributed Trace Export, which ships the router's own spans to an OpenTelemetry collector over OTLP.

Quick Start¶

1. Enable Metrics¶

Metrics are enabled by default. The metrics endpoint is available at /metrics:

# View metrics
curl http://localhost:9090/metrics

2. Configure Prometheus¶

Add the router as a target in your prometheus.yml:

scrape_configs:
  - job_name: 'continuum-router'
    static_configs:
      - targets: ['localhost:9090']
    scrape_interval: 15s

3. Import Grafana Dashboard¶

Import the provided dashboard from monitoring/grafana/dashboards/router-overview.json.

Configuration¶

Metrics configuration is done through the main config file:

metrics:
  enabled: true
  port: 9090
  path: /metrics
  max_model_labels: 1000
  max_backend_labels: 100
  cardinality_limits:
    max_models: 1000
    max_endpoints: 100
    max_error_types: 100
    enable_sampling: false
    sampling_rate: 1.0
  # External Prometheus is the durable store for this profile.
  persistence:
    enabled: false

Environment Variables¶

You can also configure metrics using environment variables:

# Enable/disable metrics
METRICS_ENABLED=true

# Change metrics endpoint
METRICS_ENDPOINT=/custom/metrics

# Enable optional metrics
METRICS_ENABLE_BODY_SIZE=true

Available Metrics¶

HTTP Metrics¶

The HTTP middleware reports backend="unknown" because backend selection happens inside handlers and not every route selects a backend. Per-backend traffic and load are available through the backend and prefix-routing metric families; client-supplied headers are never trusted for metric attribution.

Error and Retry Metrics¶

Circuit Breaker Metrics¶

Populated by ordinary proxy traffic when the metrics feature and a circuit_breaker are both enabled.

Request Parameter Policy Metrics¶

Both families use closed, bounded labels. source is one of none, local, hub, or intersection; neither family exports request values, model ids, tier ids, key ids, or policy cursors.

Backend Metrics¶

Engine Statistics Metrics¶

Emitted only with engine_stats.enabled: true, and only for engine-stats-capable backend types (vllm, sglang, llamacpp, mlxcel, ollama, lmstudio). The poller reads each engine's own load or metrics endpoint (vLLM GET /metrics, SGLang GET /v1/loads with /get_load and /metrics fallbacks, llama.cpp GET /slots and GET /metrics selected via the /props capability booleans, Ollama GET /api/ps, LM Studio GET /api/v0/models) and normalizes the answer into one snapshot per backend.

Three representation rules keep the series honest:

• Absent, not zero. A series for a figure the engine has no concept of is not emitted at all, decided statically per adapter. The one exception is SGLang /v1/loads, whose wire format omits zero-valued fields (msgspec omit_defaults); the adapter reads a missing number there as a truthful 0, so an idle SGLang backend keeps reporting backend_engine_requests_running 0 instead of vanishing.
• A failing scrape makes one statement. On any scrape error the backend emits only backend_engine_stats_scrape_success 0; the other series disappear until a scrape succeeds, so a broken scrape can never be read as an idle engine. The router keeps the last snapshot internally (visible with growing staleness_seconds on the Admin API).
• Multi-rank aggregation. An SGLang data-parallel engine reports one entry per DP rank; ranks merge into one snapshot by summing counts, capacities, and throughput and taking the maximum of usage fractions and hit rates (the saturated rank is the one that matters). The same rule applies when a Prometheus body carries multiple label sets for one series.

The two *_tokens_total counters apply reset detection (an engine value below the last observed one counts as a restart, and the new value is the delta since the restart). Reset detection is applied only to series the engine declares as counters; llama.cpp's /metrics mixes counters with gauges, and a gauge legitimately decreases.

A backend removed by hot reload loses its backend_engine_* series on the next scrape; a backend added by hot reload starts reporting within one polling interval. See engine_stats in config.yaml.example for the configuration surface, and GET /admin/backends/{name}/engine-stats for the JSON snapshot.

An engine's answer is treated as untrusted input, because a serving engine can be buggy or compromised independently of the router:

• Confined reach. The poller fetches only the configured backend URL, or a metrics_url that resolves to the same host, and it follows no redirects. The same-host rule is checked at config load and again immediately before each fetch, and it fails closed when the backend URL does not parse. A metrics_url may not downgrade an https backend to plaintext http, because the backend API key is sent with the metrics fetch; allow_external_metrics_url widens the host rule and does not lift that one.
• Nothing from the body reaches a log. Parse failures report the structural classification and the position, never the offending value, so no slice of an engine body (notably the llama.cpp /slots body) can reach the router log through a failed scrape. Transport failures report a credential-free URL.
• Bounded intake. The response is read against max_body_bytes chunk by chunk, the Prometheus parser caps samples per body and labels per sample, and the only engine free text a snapshot keeps (model names and the engine version) is length-bounded and stripped of invisible control characters. Numeric aggregation saturates instead of overflowing.

Model Service Metrics¶

Diagnosing a slow model list¶

model_refresh_duration_seconds tells you that an aggregation was slow; model_backend_fetch_duration_seconds tells you which backend made it slow, because the model-list fan-out is concurrent and the total is therefore the slowest single backend.

# Which backend dominates the refresh, 95th percentile over 5 minutes
histogram_quantile(0.95, sum by (backend, le) (rate(model_backend_fetch_duration_seconds_bucket[5m])))

Both histograms bucket up to 60 seconds. A backend exceeding the per-attempt request_timeout also emits one WARN log line naming the backend, its duration, and its attempt count, so the same finding is available without Prometheus and without raising the global log level to DEBUG:

WARN Slow model fetch from backend backend="claude-bedrock" duration_ms=15503 attempts=3 threshold_ms=5000 outcome="error"

model_backend_fetch_duration_seconds carries one series per backend name. Backend names are operator-defined and bounded by the configuration, so cardinality is normally small; a deployment that registers backends dynamically (AppProxy replicas) should expect one series per distinct replica name.

Routing telemetry is exported by the live KV cache and smart routing families below. Unsupported members of the former generic routing and model-service collectors are not registered.

Active streaming response bodies contribute to http_active_connections. The router does not currently export generic streaming duration or post-response error metrics; mid-stream fallback has the dedicated live metrics below.

Mid-Stream Fallback Metrics¶

These metrics are emitted when the mid-stream fallback feature is enabled (streaming.mid_stream_fallback.enabled: true).

Reason Label Values for streaming_fallback_total¶

Key PromQL Queries¶

# Mid-stream fallback rate
rate(streaming_fallback_total[5m])

# Fallback recovery success rate
sum(rate(streaming_fallback_success_total[5m])) /
sum(rate(streaming_fallback_total[5m]))

# Median accumulated tokens at fallback trigger
histogram_quantile(0.5, rate(streaming_fallback_accumulated_tokens_bucket[5m]))

Responses Bridge Metrics¶

Counts /v1/chat/completions requests the router dispatched to the upstream /v1/responses endpoint instead. Always registered; the counter simply stays at zero when nothing is bridged.

Reason Label Values for responses_bridge_total¶

The two mean different things operationally. responses_only_model is a standing property of the deployment and its rate tracks traffic to those models. tools_with_reasoning is a property of the traffic itself: it tells you how much of your agent traffic is paying the translation round-trip, and it drops to zero on its own once OpenAI lifts the restriction and the metadata flag is cleared.

# Share of bridged traffic that is the conditional tools bridge
sum(rate(responses_bridge_total{reason="tools_with_reasoning"}[5m])) /
sum(rate(responses_bridge_total[5m]))

Model Experiment Metrics¶

Per-variant attribution for model experiments. A dedicated family, so existing request, latency, and token series keep their label sets. Label values come from model_experiments configuration (at most 32 experiments with 16 variants each) or from closed sets.

Fallback Metrics¶

Response Cache Metrics¶

The eviction counter is exact for the in-memory LRU store. Redis and S3 perform expiry or eviction server-side and do not expose an eviction event to the router, so those backends do not fabricate a value.

Redis Cache Backend Metrics¶

These metrics are populated when the Redis cache backend is active (backend: redis).

KV Event Consumer Metrics¶

These metrics are populated when KV event consumers are active (src/infrastructure/kv_index/). vLLM, SGLang, and TensorRT-LLM listeners may use router salt echo to preserve the same prefix identity without detokenizing every block event. All backend label values are sanitized to prevent cardinality explosion.

Prefix Routing Metrics¶

These metrics track prefix-aware sticky routing decisions and backend distribution.

continuum_prefix_routing_requests_total is recorded once per backend selection and only when the PrefixAwareHash strategy arm ran, so it stays at zero under any other selection_strategy. The three labels map to the ring walk directly:

• prefix_hash: the prefix-hashed owner took the request, either because it was under the CHWBL load cap or because every backend was at the cap and the full-lap fallback kept the owner to preserve prefix locality.
• overflow: the owner was at or over the load cap and a later node clockwise on the ring took the request.
• fallback: the request carried no prefix key, so the model name was hashed and the ring was walked with no load cap. A steady 100% fallback rate almost always means prefix_routing.enabled is false.

unknown is a sanitizer output, not a routing path: it appears only if a label outside the allowlist ever reaches the recorder.

Key PromQL Queries¶

# Prefix routing hit rate (% of requests using prefix hash vs fallback)
sum(rate(continuum_prefix_routing_requests_total{strategy="prefix_hash"}[5m])) /
sum(rate(continuum_prefix_routing_requests_total[5m]))

# Overflow rate (CHWBL load balancing activations)
rate(continuum_prefix_routing_requests_total{strategy="overflow"}[5m])

# Backend load distribution (should be roughly even)
continuum_prefix_routing_backend_distribution

KV Cache Index Metrics¶

These metrics track the KV cache index subsystem including index state, query performance, routing decisions, and overlap scoring.

routing_engine_load_decisions_total is recorded once per scoring pass while routing.engine_load.enabled holds (plus once per refused selection when the admission hint trips): engine_load means the engine-load term actively ranked candidates and backend names the engine-preferred one, stale_fallback means no candidate had a fresh engine snapshot so the pass matched the base scoring (backend="none"), hysteresis_hold means fresh data existed but the waiting_requests spread stayed inside the balance thresholds, and admission_reject means the saturation admission hint refused selection because every healthy candidate reported kv_cache_usage above the threshold (backend="none"). Backend label values pass through the same sanitizer as every backend-labeled series. The gate, load normalization, cached result, and backend label are scoped to the request's exact live candidate set, so an engine excluded by model visibility, health, retry state, or per-key permissions cannot influence or be named by that scoring decision.

Both routing labels are recorded at the same seam, once per selection: kv_aware when the KV overlap scorer won and its backend was forwarded to, and fallback when the scorer ran with a prefix key but no backend cleared the pool's compiled-in combined-score threshold of 0.3, so the configured selection_strategy decided instead. A selection the scorer never attempted (no scorer registered, or no prefix key) records neither, so the two counters sum to the number of scored selections rather than to total traffic. The histogram observes the winning combined score of a kv_aware decision, which is the full weighted sum of overlap, load, and health, not the overlap term on its own.

Key PromQL Queries¶

# KV-aware routing ratio
sum(rate(continuum_kv_index_routing_decisions_total{decision="kv_aware"}[5m])) /
sum(rate(continuum_kv_index_routing_decisions_total[5m]))

# Average overlap score for routed requests
histogram_quantile(0.5, rate(continuum_kv_index_overlap_score_bucket[5m]))

# KV index query P99 latency
histogram_quantile(0.99, rate(continuum_kv_index_query_latency_seconds_bucket[5m]))

# Event source connection health
continuum_kv_index_event_source_status{status="connected"}

Smart Routing Metrics¶

These metrics cover the smart routing pipeline, including the LLM-based classifier.

Classification and Routing¶

Load Management¶

LLM Classifier¶

Aggregate and Operational¶

Key PromQL Queries¶

# Smart routing request rate by policy
rate(smart_routing_requests_total[5m])

# Tier usage distribution
sum by(tier) (rate(smart_routing_tier_usage_total[5m]))

# LLM classifier cache hit rate
rate(smart_routing_llm_classifier_cache_hits_total[5m]) /
rate(smart_routing_llm_classifier_calls_total[5m])

# LLM classifier P95 latency
histogram_quantile(0.95, rate(smart_routing_llm_classifier_duration_seconds_bucket[5m]))

# LLM classifier fallback rate (reliability indicator)
rate(smart_routing_llm_classifier_fallbacks_total[5m]) /
rate(smart_routing_llm_classifier_calls_total[5m])

# Fraction of requests classified by LLM vs rule-based
rate(smart_routing_classifications_total{classifier_type="llm_based"}[5m]) /
rate(smart_routing_classifications_total[5m])

# Policy evaluation success rate
sum by(policy_name) (rate(smart_routing_policy_evaluations_total{result="matched"}[5m]))

Business Metrics¶

Guardrail Metrics¶

Exported when guardrails are configured and the metrics feature is enabled. Every guardrail decision is recorded so operators can observe what a policy does (or would do, in monitor mode) before and after enforcement.

Label values:

• stage is input, output, or streaming.
• result is allow, block, transform, or flag.
• kind is timeout or error.
• mode is monitor or enforce. Because guardrail_verdicts_total carries mode, monitor-mode verdicts are visible even though they never gate a request, which is what makes the monitor-then-enforce rollout observable.
• provider is a configured provider name, or one of the two reserved match-list labels: match_list_deny when a guardrails.deny rule decided the check, match_list_allow when an allow rule did. A deny-list block also carries category="deny_list" on guardrail_blocks_total. See Allow and deny lists.
• strategy is the strategy in effect when the cap tripped (buffer_full / chunked / monitor); outcome is how output checking continued past it (degraded_chunked / compacted / truncated). See the buffer cap section for what each outcome means for the safety guarantee. strategy="monitor" is not a safety event: it means a monitor-mode stream was long enough that only its first 4 MiB was observed.

Key PromQL Queries¶

# What would be blocked, broken down by category (monitor-mode tuning)
sum by (category) (rate(guardrail_blocks_total[1h]))

# Block rate per stage after enforcement
sum by (stage) (rate(guardrail_verdicts_total{result="block", mode="enforce"}[5m]))

# Provider error rate (timeouts vs hard failures)
sum by (provider, kind) (rate(guardrail_errors_total[5m]))

# P95 guardrail check latency per provider
histogram_quantile(0.95, sum by (le, provider) (rate(guardrail_check_duration_seconds_bucket[5m])))

For the full guardrail guide (concepts, providers, configuration, and the threshold-tuning workflow), see Guardrails.

Per-API-Key LLM Token Usage¶

The router publishes a per-API-key breakdown of LLM token consumption so operators can answer questions like "which key consumed the most completion tokens last hour?" or "how many prompt tokens did team X spend on model Y today?". This data is independent of the model_tokens_processed aggregate counter and supports capacity planning, fair-use enforcement, and external cost attribution.

Metric Definition¶

kind is one of:

• prompt — tokens in the upstream request prompt
• completion — tokens in the upstream response completion

Both OpenAI-compatible (prompt_tokens / completion_tokens) and Anthropic (input_tokens / output_tokens) response shapes are normalized into the same counter. The router also injects stream_options.include_usage=true on OpenAI-compat streaming requests so usage data arrives in the final SSE chunk regardless of client behavior.

api_key_id Derivation¶

api_key_id is never the raw API key. The router derives a stable, non-reversible identifier in this priority order:

1. If the request's bearer token matches a configured API-key entry, the entry's id field is used (e.g., key-production-1).
2. Otherwise, the router computes SHA-256 over the raw token and uses the first 12 hex characters prefixed with k_ (e.g., k_3f5a7c9b1e2d).
3. If no token is presented, the literal value anonymous is used.

All label values flow through the existing CardinalityManager so a runaway/rotating-key attack cannot exhaust Prometheus series.

Annotation Labels and api_key_info¶

Each configured API key may carry an optional free-form annotations: { key: value } map. Operators declare which annotation keys become Prometheus labels via the global metrics.annotation_labels allowlist; everything else stays internal.

Configuration schema (under the existing api_keys block):

api_keys:
  api_keys:
      - key: "${API_KEY_1}"
        id: "key-production-1"
        user_id: "user-admin"
        organization_id: "org-main"
        annotations:
            email: "ops@example.com"
            team: "platform"
            environment: "prod"
            owner: "alice"

metrics:
    enabled: true
    annotation_labels: [email, team]            # Allowlist of label keys

Reserved annotation keys (recommended canonical names, not enforced): email, uuid, owner, team, environment. Operators may add custom keys.

When metrics.annotation_labels is non-empty, the router publishes api_key_info{api_key_id, email, team, ...} = 1 once per known key. Use PromQL joins to project the metadata onto llm_tokens_total without bloating its label set:

# Tokens per email (sums prompt + completion, last 24h)
sum by (email) (
  increase(llm_tokens_total[24h])
  * on (api_key_id) group_left(email) api_key_info
)

Cardinality and Hot-Reload¶

• api_key_id cardinality is bounded at 1000 by default.
• Hot-reload of API-key annotations is supported via the existing config-reload pipeline. The api_key_info info-metric is republished atomically on every reload; counter values for llm_tokens_total are never reset.
• The label set on api_key_info (i.e., the contents of annotation_labels) is frozen at startup. Adding or removing keys from the allowlist requires a restart — Prometheus does not allow renaming labels on a registered metric.

Example PromQL Queries¶

# Total prompt tokens consumed per API key in the last hour
sum by (api_key_id) (
  increase(llm_tokens_total{kind="prompt"}[1h])
)

# Top 10 keys by completion tokens in the last 24h
topk(10,
  sum by (api_key_id) (
    increase(llm_tokens_total{kind="completion"}[24h])
  )
)

# Tokens grouped by team (requires team in annotation_labels)
sum by (team) (
  increase(llm_tokens_total[24h])
  * on (api_key_id) group_left(team) api_key_info
)

# Combined prompt+completion rate per model (tokens/sec)
sum by (model) (rate(llm_tokens_total[5m]))

# Per-key consumption by backend (useful for cost attribution)
sum by (api_key_id, backend) (
  increase(llm_tokens_total[24h])
)

Grafana Panel Example¶

A simple Grafana stat panel showing the top 10 teams by completion tokens over the last 24 hours:

{
    "title": "Top 10 teams by completion tokens (24h)",
    "type": "stat",
    "targets": [
        {
            "expr": "topk(10, sum by (team) (increase(llm_tokens_total{kind=\"completion\"}[24h]) * on (api_key_id) group_left(team) api_key_info))",
            "legendFormat": "{{team}}"
        }
    ],
    "options": {
        "reduceOptions": {
            "values": false,
            "calcs": ["lastNotNull"]
        }
    }
}

For tracking spend trends, pair this with a time-series panel using rate(llm_tokens_total[5m]) grouped by team or model.

Verification Steps¶

After enabling the feature:

1. Issue a chat-completion request with a configured API key.
2. Scrape /metrics and confirm llm_tokens_total{...} and api_key_info{...} series appear.
3. For streaming, verify the counter still increments — usage is captured from the final SSE chunk. The router injects stream_options.include_usage=true automatically for OpenAI-compat backends so this works regardless of client behavior.
4. Inspect /metrics cardinality on a typical workload (e.g., wc -l < /metrics) to confirm no regression versus the prior baseline.

Integration¶

Prometheus Configuration¶

Complete Prometheus configuration example:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'continuum-router'
    static_configs:
      - targets: ['router1:9090', 'router2:9090']
    metric_relabel_configs:
      # Drop high-cardinality metrics if needed
      - source_labels: [__name__]
        regex: 'http_request_duration_seconds_bucket'
        action: drop

Kubernetes Integration¶

For Kubernetes deployments, use ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: continuum-router
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: continuum-router
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics

The Helm chart can render this object with serviceMonitor.enabled=true when the Prometheus Operator CRD is installed. For clusters without the operator, the repository also ships a standalone Prometheus Kustomize bundle with Kubernetes endpoint discovery and the checked-in alert rules:

kubectl apply -k monitoring/prometheus
kubectl -n monitoring rollout status deployment/continuum-router-prometheus
kubectl -n monitoring port-forward service/continuum-router-prometheus 9090:9090

The standalone profile requests a 10 GiB ReadWriteOnce PersistentVolumeClaim with 15-day retention. Confirm the default StorageClass and resize the claim for expected series volume, or use a managed Prometheus service for highly available monitoring. Review its cluster-wide discovery RBAC and NetworkPolicy before applying it.

Grafana Dashboard¶

monitoring/grafana/dashboards/router-overview.json contains these panels:

• Request rate
• Error rate
• P95 latency
• Backend health
• Request rate by endpoint
• Response-time percentiles
• Model usage distribution
• Current in-flight load across backends
• Engine requests (running / waiting), from engine_stats (issue #1446)
• Engine KV cache usage, from engine_stats (issue #1446)
• Engine-load routing decisions by reason, from routing.engine_load (issue #1447)

To import it:

1. Open Grafana.
2. Select Dashboards → Import.
3. Upload monitoring/grafana/dashboards/router-overview.json.
4. Select the Prometheus data source and import.

Alerting¶

Pre-configured alert rules are available in monitoring/prometheus/alerts.yml:

Critical Alerts¶

{% raw %}

- alert: HighErrorRate
  expr: sum(rate(http_requests_total{status_code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
  for: 5m
  annotations:
    summary: "High error rate: {{ $value | humanizePercentage }}"

Warning Alerts¶

{% raw %}

- alert: HighLatency
  expr: histogram_quantile(0.95, http_request_duration_seconds) > 1
  for: 5m
  annotations:
    summary: "P95 latency above 1s: {{ $value | humanizeDuration }}"

- alert: TimeoutErrors
  expr: sum(rate(timeout_errors_total[5m])) > 0.1
  for: 10m
  annotations:
    summary: "Frequent timeout errors: {{ $value }} per second"

Examples¶

Query Examples¶

Request Rate by Status¶

sum(rate(http_requests_total[5m])) by (status)

P95 Latency by Endpoint¶

histogram_quantile(0.95, 
  sum(rate(http_request_duration_seconds_bucket[5m])) by (endpoint, le)
)

Backend Load Overview¶

backend_current_load

Model Usage Ranking¶

topk(10, sum(rate(model_usage_total[1h])) by (model))

Error Rate Percentage¶

sum(rate(http_requests_total{status_code=~"5.."}[5m])) /
sum(rate(http_requests_total[5m])) * 100

Programmatic Access¶

You can also access metrics programmatically:

import requests
from prometheus_client.parser import text_string_to_metric_families

# Fetch metrics
response = requests.get('http://localhost:9090/metrics')
metrics = text_string_to_metric_families(response.text)

# Process metrics
for family in metrics:
    for sample in family.samples:
        if sample.name == 'http_requests_total':
            print(f"Endpoint: {sample.labels['endpoint']}, Count: {sample.value}")

Custom Metrics Collection¶

#!/bin/bash
# Collect metrics every 30 seconds and save to file

while true; do
  timestamp=$(date +%s)
  curl -s http://localhost:9090/metrics > "metrics_${timestamp}.txt"
  sleep 30
done

Best Practices¶

1. Label Cardinality¶

Keep label cardinality low to prevent metric explosion:

# Good: Low cardinality
labels:
  status: "200"  # ~5 possible values
  method: "GET"  # ~7 possible values

---

# Bad: High cardinality
labels:
  user_id: "12345"  # Unbounded
  request_id: "abc-123"  # Unique per request

2. Metric Naming¶

Follow Prometheus naming conventions:

• Use snake_case
• Include units in metric names (_seconds, _bytes, _total)
• Use standard prefixes (http_, backend_, model_)

3. Dashboard Design¶

• Group related metrics together
• Use appropriate visualization types (gauge for current values, graph for time series)
• Include both absolute values and rates
• Set reasonable refresh intervals (15-30s for real-time, 1-5m for historical)

4. Alert Configuration¶

• Use appropriate evaluation periods (for: 5m to avoid flapping)
• Include context in alert descriptions
• Set up alert routing based on severity
• Test alerts in staging before production

5. Performance Considerations¶

• Disable optional metrics if not needed
• Use recording rules for complex queries
• Implement proper metric retention policies
• Consider using remote storage for long-term retention

6. Security¶

• Protect metrics endpoint if sensitive data is exposed
• Use TLS for Prometheus scraping in production
• Implement authentication for Grafana dashboards
• Audit metric access logs

Troubleshooting¶

Metrics Not Appearing¶

1. Check if metrics are enabled in configuration
2. Verify the metrics endpoint is accessible
3. Check Prometheus target status
4. Review router logs for metric initialization errors

High Memory Usage¶

1. Review cardinality limits
2. Check for unbounded labels
3. Reduce histogram buckets if needed
4. Enable metric expiration

Incorrect Values¶

1. Verify metric types (counter vs gauge)
2. Check aggregation functions
3. Review label selectors
4. Validate time ranges

Additional Resources¶

• Prometheus Documentation
• Grafana Documentation
• Prometheus Best Practices
• Monitoring Distributed Systems