Distributed Trace Export (OTLP)¶

Continuum Router can export its own spans to an OpenTelemetry collector over OTLP. Exported spans describe how a request moved through the router: which route it hit, which backend was selected under which strategy, how many attempts it took, and how each attempt ended. They carry no prompt text, no completion text, and no credentials.

This page covers what the feature adds on top of the trace-context propagation the router has always done, how to build and configure it, what exactly is exported, and how the privacy boundary is enforced.

What this is, and what it is not¶

The router has always propagated trace context. It extracts a trace ID from the inbound request, generates one when the client sent none, and forwards X-Request-ID, X-Trace-ID, X-Correlation-ID, and the W3C traceparent / tracestate pair to backends. That behavior is described in Distributed Tracing and is unrelated to this page.

What is new is span export: the router now emits OpenTelemetry spans of its own and ships them to a collector.

What export does not change¶

• With tracing.otlp absent, or present with enabled: false, inbound and outbound trace-context behavior is byte for byte what it was before export existed. The same headers are read, the same headers are written, with the same values.
• Export never participates in the request path's success or failure. A collector that is down, unreachable, or rejecting exports produces log warnings and dropped spans, never a failed or delayed client request.
• Export is not a metrics replacement. Rates, latencies, and error counts still come from the Prometheus endpoint described in Metrics and Monitoring.

Build requirement¶

Export lives behind the otel Cargo feature, which is not part of default or full. A build without it never resolves opentelemetry, opentelemetry_sdk, opentelemetry-otlp, or tracing-opentelemetry at all.

cargo build --release --features otel

Official release binaries are compiled with otel. The runtime switch tracing.otlp.enabled still defaults to false, so export is opt-in twice over: once at build time, once in configuration.

Behavior of a binary built without the feature¶

A default build still parses and validates the whole tracing.otlp section exactly as an otel build does, so one config.yaml stays valid across build variants and continuum-router config validate gives the same answer either way. If such a binary starts with tracing.otlp.enabled: true, startup logs a warning that span export is inactive and continues serving traffic normally. An inert observability setting is not a reason to refuse to route.

Configuration¶

Every field below is shown with its real default, so the block is equivalent to omitting the section apart from enabled.

tracing:
  enabled: true                              # Unrelated to export: header propagation
  w3c_trace_context: true

  otlp:
    enabled: false                           # Default: false. Install the exporter at startup
    endpoint: "http://localhost:4318/v1/traces"  # http: FULL traces path. grpc: base endpoint
    protocol: http                           # Default: http. One of: http, grpc
    timeout: "10s"                           # Default: 10s. Bounds each export request

    # Sent with every export request. Values support ${ENV_VAR} and
    # ${ENV_VAR:-default}. Never write a literal secret here.
    headers:
      Authorization: "Bearer ${OTLP_AUTH_TOKEN}"

    sampling:
      ratio: 0.05                            # Default: 0.05. Fraction of root traces, 0.0 to 1.0
      parent_based: true                     # Default: true. Honor an inbound sampling decision

    batch:
      max_queue_size: 2048                   # Default: 2048. Spans buffered before new ones drop
      max_export_batch_size: 512             # Default: 512. Spans per export request
      scheduled_delay: "5s"                  # Default: 5s. Interval between scheduled flushes

    resource:
      service_name: "continuum-router"       # Default: "continuum-router"
      service_namespace: null                # Default: unset. Sets service.namespace
      deployment_environment: null           # Default: unset. Sets deployment.environment.name
      attributes: {}                         # Default: empty. Extra string resource attributes

    shutdown_timeout: "5s"                   # Default: 5s. Bounds the final flush on shutdown

Secrets¶

Header values must never contain a literal credential in a committed configuration file. Use ${ENV_VAR} substitution, the same dialect api_keys, web_search, and control_plane use, and supply the value through the environment:

export OTLP_AUTH_TOKEN="..."

Validation deliberately runs against the unresolved template, so continuum-router config validate gives the same verdict on a developer laptop as in the deployment, and no secret has to be present to check a config file. Substitution happens once, at exporter construction. A header whose value resolves to an empty string is dropped rather than sent empty. A ${VAR} with no :-default whose variable is unset is left in place literally and logged as a warning naming the variable, never the value.

Field reference¶

The router also attaches service.version from its own crate version to every exported span. You do not set it.

Resource attributes describe the deployment, not the traffic. Keep them to low-cardinality deployment metadata such as region or cluster, and never put per-request or per-tenant identifiers there.

HTTP and gRPC endpoints¶

The two protocols disagree about what an endpoint is, and the router validates each accordingly rather than guessing.

protocol: http¶

The endpoint is the full traces path. A bare host is rejected at validation time with a message naming the expected form:

otlp:
  protocol: http
  endpoint: "http://collector.example.com:4318/v1/traces"   # correct

otlp:
  protocol: http
  endpoint: "http://collector.example.com:4318"             # rejected: missing traces path

Transport reuses the router's existing async reqwest client, which is rustls only. There is no blocking HTTP client and no OpenSSL in the graph.

protocol: grpc¶

The endpoint is the base endpoint, with no path:

otlp:
  protocol: grpc
  endpoint: "https://collector.example.com:4317"

An https gRPC endpoint uses rustls with the ring provider and the bundled webpki root certificates. The OS certificate store is not consulted. A collector presenting a certificate from a private CA that is not in the webpki set will therefore fail verification even though curl on the same host succeeds.

Configured headers become gRPC metadata. Metadata keys must be lowercase ASCII, so keys are lowercased on the way out; a header that still cannot be converted is dropped with a warning that names the header but never its value.

Sampling¶

sampling.ratio defaults to 0.05, meaning one root trace in twenty is sampled. The default sits well below 1.0 on purpose. Full sampling on a router fronting production inference traffic is a straightforward way to make the collector, rather than the backends, the bottleneck. Raise it deliberately, and prefer raising it in a staging deployment first.

sampling.parent_based defaults to true, which is what makes the router a well-behaved participant in a trace that started upstream:

• When an inbound traceparent says the trace is sampled, the router's spans are recorded regardless of ratio, so the upstream trace is not left with a hole where the router should be.
• When an inbound traceparent says the trace is not sampled, the router records nothing for it, so an unsampled trace does not acquire a disconnected fragment.
• When there is no inbound traceparent, the router is the root and ratio decides.

Setting parent_based: false applies ratio to every trace, including ones with a sampled parent. That produces traces with the router's contribution missing, and is only reasonable when the router is deliberately treated as an independent trace source.

Batching and backpressure¶

Finished spans go into a bounded queue and are exported in batches by a background processor.

The processor drops spans when the queue is full rather than blocking the thread that produced them. This is the deliberate trade at the center of the design: losing spans is recoverable, stalling inference traffic is not. A slow, wedged, or unreachable collector can never add latency to a proxied request.

There is no per-batch export timeout¶

That knob exists in the OpenTelemetry SDK only under an experimental async-runtime processor, and a setting that silently does nothing is worse than a setting that does not exist. It is therefore deliberately absent from the configuration surface. tracing.otlp.timeout bounds each individual export request, which is the bound that actually matters when a collector stops answering.

Shutdown¶

During graceful shutdown the router stops producing new spans, then flushes what is already queued, bounded by shutdown_timeout (default 5s).

Spans still buffered when that budget expires are abandoned. Shutdown proceeds, and a warning is logged saying so. Draining a backlog into a slow collector is not worth extending a shutdown window that a supervisor or an orchestrator is timing.

Exported spans and events¶

The exported surface is a fixed, bounded taxonomy. Nothing outside the tables below is ever sent.

Spans¶

router.request is created by the request middleware and every other span nests under it. router.select_backend covers strategy evaluation and the resulting pick. router.backend_call covers one outbound attempt, so a request that retried produces several, distinguished by router.attempt.

Span events¶

router.retry is emitted for every retryable failure, including one on the final attempt, so a trace never ends without an explanation. router.circuit_breaker is emitted from the single breaker transition point, so every path that moves a breaker produces exactly one event.

Attribute values¶

Three OpenTelemetry control fields (otel.name, otel.kind, otel.status_code) also travel with each span. They set the span name, kind, and status rather than carrying data.

Privacy: what is never exported¶

Exported spans are metadata only. The following never appear in an exported span, an attribute, an event, or a status message:

• Prompt text and completion text
• Request bodies and response bodies, in whole or in part
• Provider error bodies and error messages
• API keys, bearer tokens, and provider credentials
• Raw request paths and query strings

Failure information reaches a span only through a classifier that maps a router error onto one of the fixed strings listed above. The message, the backend-authored body, and every error payload are discarded at that point, so there is no path by which a provider's error text becomes a span attribute.

Two enforcement layers¶

The guarantee does not rest on reviewers noticing a careless tracing::info_span!(prompt = %body) somewhere in the proxy path. It is enforced twice, independently.

1. At construction. Every span and event the router creates comes from a constructor in a single module, built from a fixed set of static field names. There is no constructor that accepts arbitrary key/value pairs, so there is no in-tree way to attach an unplanned field.
2. At the export boundary. A span processor runs after conversion and before the exporter, and sees everything that would be sent regardless of which code produced it. It drops:
   • any span whose name is not one of the three in the taxonomy, which removes instrumentation emitted by dependencies and by unrelated parts of the router;
   • any attribute whose key is not on the allow-list, removed rather than redacted, so no residue of the value or even its length survives;
   • any span event that is not one of the router's own two events.

That third rule is what keeps ordinary log lines out. A tracing log record emitted inside an instrumented span becomes a span event whose message attribute holds the formatted log line, and that line can contain a provider error body. Dropping such events by name at the boundary is the control that prevents it.

The free-form OpenTelemetry status message is also deliberately outside the allow-list. The router never sets it, and any error status description that reaches the boundary is cleared.

A source-level test asserts that the taxonomy module and the allow-list stay in sync, so adding a field without adding it to the allow-list fails the build, as does naming a field anything that looks like message content or a credential.

W3C Trace Context interoperability¶

When an inbound request carries a valid traceparent, the exported router.request span is parented from it. The router therefore joins the upstream trace rather than starting a disconnected one, and a caller that instruments its own client sees the router's spans nested under its own.

Outbound traceparent injection to backends is unchanged by export. The router writes the same headers with the same values it always has, whether or not tracing.otlp is enabled.

Malformed inbound headers leave the span as a trace root. The router's existing trace-context validation already rejects and logs those, so the export path stays silent rather than warning twice per request.

The W3C propagator itself is installed only when export is installed. Before that, the OpenTelemetry global propagator is the SDK no-op, which is why a router without export has no OpenTelemetry-side behavior to speak of.

Cost when export is off¶

The request path checks one relaxed atomic load and, when export is not active, forwards the request untouched: no header is read for OpenTelemetry purposes, no configuration is cloned, and no tracing span machinery is entered. That is the state of every default build and of every otel build that has not set tracing.otlp.enabled: true.

Span constructors are guarded by the same flag and return a disabled span, which also makes the later record_* calls no-ops, so call sites need no second guard.

One cost is specific to otel builds and applies whether or not export is enabled: the subscriber carries a reloadable layer slot so the OpenTelemetry layer can be swapped in after configuration is loaded, and a reloadable layer cannot use tracing's static per-callsite interest cache. That costs a lock read per callsite evaluation. It does not exist in a default build.

Library embedders¶

Embedders that install their own tracing subscriber (see Library Usage) do not get the reloadable slot. Export reports that no layer slot is installed and stays off; such embedders must add the tracing-opentelemetry layer themselves.

Troubleshooting¶

See also¶

• Distributed Tracing for header propagation and correlation IDs, which are independent of export
• Metrics and Monitoring for Prometheus metrics
• Persistent Metrics Log for restart-surviving local metric history
• Circuit Breaker for the state machine behind the router.circuit_breaker event