LLM Observability Callbacks: Design Record¶

This note records the design of the first-party callback integrations that translate per-request usage into an LLM observability platform's native schema (issue #2376). Langfuse ships today. The page describes the implemented contract, not a plan.

Today¶

The observability surface is OTLP tracing (tracing.otlp, --features otel), Prometheus metrics, the process log, and the usage export (/admin/usage/export). The observability.callbacks section adds a fifth surface: one trace per request, mapped onto the platform's native object model, so the operator no longer runs and maintains a translation component downstream of OTLP or the usage export.

The section is independent of tracing.enabled. Turning distributed tracing off never silently disables callback export, and enabling callbacks does not turn on OTLP.

Configuration¶

observability:
  callbacks:
    enabled: true
    include_content: false
    max_queue: 1024
    max_batch_records: 32
    sinks:
      - provider: langfuse
        endpoint: https://cloud.langfuse.com
        public_key_env: LANGFUSE_PUBLIC_KEY
        secret_key_env: LANGFUSE_SECRET_KEY
        allow_private_endpoint: false
        timeout: 5s
        max_attempts: 3
        initial_delay: 500ms
        max_delay: 30s

The section is optional and off by default. With no observability: section nothing is queued, sent, or logged, and the request path pays one OnceLock lookup and one atomic load.

The config types and the runtime are compiled under every Cargo feature set, so a configuration file validates identically everywhere and no build silently drops the section. When enabled is true, the section requires at least one sink. max_queue is fixed at startup and needs a restart to change; the remaining fields apply on the next published reload.

The provider set and the credential variables resolve at apply time. A sink whose credentials are missing in the environment becomes inert with a warning that names the variable and never its value; the other sinks keep delivering.

Event sources¶

One request yields exactly one trace from two producers joined by one per-request flag:

• Content-capable. Prompt-log capture calls the callback runtime after its redaction pass with the assembled record. This covers /v1/chat/completions, /v1/responses, and /anthropic/v1/messages, streamed or not, because capture already writes exactly one record per request.
• Metadata-only. The completion path contributes the request-scoped facts from the request ledger (src/services/stats/, issue #2053): model, backend, token counts, latency, and outcome. It serves every other route, and the three routes above when capture did not admit the request. It skips itself when the shared flag says the content producer emitted.

A streamed request produces one trace whose output is assembled when the stream is finalized, not one event per chunk, so streamed and non-streamed requests have the same shape.

Content policy¶

Prompt, completion, and tool-call text leave the process only when the operator set include_content: true and prompt-log capture admitted the request under its per-key consent gate, so the redaction pass has already run. When include_content is false the runtime strips the content fields from the record before queueing, and a metadata-only producer never assembles them. Credentials are read from the *_env variables at startup, used only for HTTP Basic auth, and never logged, never put in an error, and never sent anywhere else. The endpoint is treated as a sensitive value in configuration output and diff.

Delivery¶

Delivery is fail-open and off the request path, mirroring the alerting runtime (src/services/alerting/). Producers only try_send into one bounded queue. A dedicated worker performs the HTTP, batches up to max_batch_records traces formed under the same configuration generation, and never blocks a response. A full queue drops the trace and counts it, once per configured sink.

Each sink retries with bounded exponential backoff (max_attempts, initial_delay, max_delay). A partial-success response counts the rejected spans as failed instead of re-sending the accepted ones. A 4xx other than 408 or 429 is permanent and is not retried, so a wrong credential or refused body cannot occupy the single worker for the whole budget. On shutdown the worker signals the queue to drain; a delivery already in flight finishes under its own timeout budget first, and the queued records then drain within five seconds.

Endpoints go through the shared EndpointPolicy (src/services/export_webhook/policy.rs): redirects are off, DNS is pinned, and loopback and private addresses are rejected unless allow_private_endpoint is set for a self-hosted collector.

Schema mapping¶

The Langfuse adapter posts to POST {endpoint}/api/public/otel/v1/traces with the x-langfuse-ingestion-version: 4 header and Basic auth, and reuses the OTLP/JSON mapper the prompt-log export already ships (src/services/prompt_log_export/langfuse.rs). One request becomes one OTLP span carrying the redacted prompt and completion, the usage numbers, and the outcome. The callback adapter binds its own scope and span names, so callback traces and prompt-log export traces never look like the same stream.

Adapters speak plain HTTP through the existing reqwest client rather than linking provider SDKs, so they add no dependency to the embed negative dependency contract.

Metrics¶

observability_callback_events_total{sink,result} counts delivered, failed, and dropped traces per sink. observability_callback_queue_depth reports the current queue occupancy.

Deferred providers¶

Braintrust, Arize, and Langsmith are follow-ups behind the same adapter seam. Each needs its own native schema mapping and authentication, so each ships as its own issue rather than behind an unused generic interface.