Log System Architecture¶

Status¶

This document defines the architecture of Continuum Router's bounded asynchronous process-log pipeline. The implementation uses restart-required logging.queue_capacity (default 8,192 records) and logging.queue_capacity_bytes (default 16 MiB), plus a fixed 64 KiB per-record ceiling and a one-second graceful-shutdown budget.

Operators can observe loss and degradation through continuum_process_log_dropped_total, continuum_process_log_write_errors_total, and continuum_process_log_truncated_total, plus the continuum_process_log_queue_depth and continuum_process_log_queue_bytes gauges.

By default, logs go to stdout or stderr so the container runtime, systemd/journald, or another supervisor can own persistence and rotation. An optional logging.file block selects a directly managed file sink with the following behavior:

• Rotation and permissions: Rotates between complete records at rotation.max_bytes, retains rotation.max_files matching files (and, with rotation.max_age_days, none older than that age), never overwrites an existing name, and creates Unix files with mode 0600.
• Filename templates: Accepts {date}, {time}, {pid}, and the required {sequence} placeholder.
• Operational constraints: Internal rotation must not be combined with an external rename-based logrotate rule for the same path because the process keeps its active descriptor open. The configured directory must also not be shared by concurrently running Router processes, because retention is directory-wide for names matching the pattern and could otherwise remove another process's segment.

The file sink writes plain text with ANSI colour codes disabled, while stdout and stderr keep the tracing-subscriber default (colour unless NO_COLOR is set).

Rotation and retention are best-effort. When a new segment cannot be created (full disk, removed directory, exhausted sequence) or the directory cannot be listed for pruning, the sink reports the problem on stderr and keeps writing to the open segment, retrying rotation after another rotation.max_bytes of output. Only a failed write to the open segment fails the pipeline.

The writer flushes at least once per second (including while continuously busy) and flushes on rotation and graceful shutdown. It does not call fsync/sync_all, issuing zero explicit fsync calls per record so logging does not become durable transaction I/O; the operating system controls page-cache writeback. A machine or kernel crash may therefore lose the most recent file-backed logs. Deployments requiring stronger log durability should use stdout with a dedicated log collector rather than adding fsync latency to the request environment.

The initial three-trial synchronous/asynchronous comparison and regression budget are recorded in Process Log Pipeline Benchmark (2026-09-19).

The priorities, in order, are:

1. Keep the Router process alive and preserve useful service capacity under sustained overload, even when that requires dropping diagnostic logs.
2. Preserve request availability and process stability when the log sink is slow, unavailable, or producing repeated write errors.
3. Preserve the diagnostic value and security properties of existing logs.
4. Remove log-sink latency from the request path.
5. Improve throughput only where measurements show that logging is a bottleneck.

Service survival is the governing rule. Process logs are expendable under pressure; request-path progress, bounded memory, health reporting, and graceful degradation are not. No logging failure may panic, abort, exhaust memory, or indefinitely occupy a Tokio worker.

Replaceable logger boundary¶

src/core/tracing/logger.rs is the implementation boundary between subscriber composition and log delivery. LogPipelineFactory constructs a pipeline, EventWriterFactory supplies one record writer per formatted event, and LogPipelineGuard owns implementation-specific shutdown. The subscriber knows only these interfaces; the bounded asynchronous writer is the default AsyncProcessLoggerFactory, not a hard-coded dependency.

Alternative synchronous, remote, test, or platform-native pipelines can be installed through init_process_subscriber_with_factory without changing the formatting layer or OTLP composition. Selection happens during process startup; replacing a globally installed tracing subscriber at runtime is not supported because callsite interest and lifecycle state are process-wide.

The default async implementation keeps a concrete enum fast path, so producing an event returns EventBuffer directly and adds no trait-object allocation. Only externally supplied dynamic writer factories pay the per-event boxing cost. LoggerGuard::status and its shutdown result expose an optional implementation-neutral LoggerStatus; custom guards may return None when they have no comparable counters.

LoggerWriter is Send + Sync and may be cloned across producer threads; each call creates an independent LoggerEventWriter, which is Send. Queue mutation is serialized only for the short push/pop critical section, while the single destination writer preserves accepted queue order. Per-producer order is therefore preserved, although interleaving between producers is scheduler-dependent.

The queue uses a short Mutex<VecDeque> push/pop section and allocates at most 1,024 slots initially. The writer never holds this mutex during destination I/O, and the queue remains bounded by the configured record and payload-byte limits.

Scope¶

This design covers the human-readable process logs emitted through tracing and formatted by tracing-subscriber. It covers HTTP server mode, stdio mode, one-off commands, and coexistence with the optional OTLP span exporter.

The following are outside the process-log delivery contract:

• billing and usage records;
• audit records that must be durably retained;
• Prometheus metrics;
• OTLP spans;
• application protocol output written to stdout by CLI, ACP, or MCP commands.

Billing, quota, and audit correctness never depend on successful delivery of a process log line. Those records use their own acknowledged and reconcilable persistence paths.

Bootstrap and OTLP independence¶

If a formatting layer writes directly to std::io::stdout or std::io::stderr, destination I/O happens while the emitting task is being polled, allowing a slow container log driver, pipe, terminal, or journal to delay a Tokio worker that is also serving requests.

The optional OTLP exporter uses its own batching design, and enabling it does not make the formatting layer asynchronous. The two output paths are independent.

Subscriber initialization happens before configuration loading because configuration loading itself emits logs. Consequently, queue settings cannot simply be read from the normal configuration and applied to the first log line. Any schema field affecting the log pipeline must resolve this bootstrap ordering explicitly rather than applying late.

Required properties¶

Availability and bounded resources¶

• A blocked log destination must not block an HTTP request handler or a Tokio runtime worker.
• The in-memory queue must have a fixed upper bound.
• Both record count and retained payload bytes must be bounded; limiting only the number of variable-sized records is not a memory bound.
• Queue saturation must not allocate an unbounded spill buffer.
• Diagnostic log loss is preferable to request-path blocking or process OOM.
• A worker failure must not panic the request path.
• Repeated destination errors must be contained within the logging subsystem; they must not trigger a retry loop that consumes unbounded CPU or emits logs recursively.
• Once the queue is saturated, producer work must remain O(1) and must not wait for queue capacity.
• Logging overload must not make liveness or readiness handling depend on the failed writer.

Degradation policy¶

The process-log path is explicitly fail-open for serving traffic:

1. When the queue has capacity, enqueue the formatted record.
2. When the queue is full, drop the new record and increment the drop counter.
3. When the writer is disconnected or permanently failing, stop attempting expensive per-event recovery and continue serving.
4. Never synchronously fall back to stdout or stderr from a request task. Such a fallback would recreate the original failure mode when the sink is unhealthy.
5. Expose the degraded state through metrics and process status that do not depend on the log writer.

This policy applies only to diagnostic process logs. A billing, usage, or audit operation that has a fail-closed business contract enforces that contract through its own persistence and admission path rather than through logging.

Observable loss¶

• Every line rejected because the queue is full must increment a monotonic dropped-line counter.
• The counter must be readable without sending another event through the same saturated log queue.
• When the metrics feature is active, the count must be exported as a low-cardinality Prometheus metric.
• Shutdown diagnostics report loss on a best-effort basis, while metric state remains the source of truth; attempting to log the report must not delay shutdown indefinitely.

Output compatibility¶

• HTTP mode continues to send process logs to stdout.
• Stdio, ACP, MCP, and one-off command modes keep protocol output on stdout and diagnostics on stderr.
• Existing field escaping, redaction, filtering, and formatting behavior is preserved unless a separate change explicitly modifies it.
• The OTLP reloadable layer and its privacy boundary continue to operate independently of the formatting writer.

Lifecycle safety¶

• The writer guard must live until process shutdown; dropping it during startup must not stop asynchronous delivery.
• std::process::exit runs no destructors, so a guard held by main never sees an exit taken inside startup code. The installed pipeline is therefore owned by core::tracing::subscriber, and every exit after installation goes through subscriber::exit_process, which drains the queue within the same finite budget before exiting. Without it, the records explaining why startup failed would remain queued when the process ends.
• Graceful shutdown gets a finite flush budget.
• Once the budget expires, shutdown proceeds and records the incomplete flush.
• Abrupt termination cannot guarantee delivery and must not be documented as if it can.
• Library embedders remain responsible for installing and managing their own subscriber.

Pipeline architecture¶

tracing event on request/background task
                 |
                 v
       existing filters and formatter
                 |
                 v
       bounded non-blocking enqueue
          | accepted       | full/disconnected
          v                v
    fixed-capacity queue   atomic dropped counter
          |
          v
   dedicated writer thread
          |
          v
 stdout or stderr destination

The implementation uses the bounded writer in src/core/tracing/process_log.rs. A tracing-appender prototype was rejected because dropping its worker guard joins the writer thread without a timeout, which can block shutdown indefinitely when the underlying Write call is stalled. The router-specific writer owns the finite flush budget and exposes its counters directly.

The default queue policy is lossy under saturation. A lossless bounded mode would block producers and violate the availability requirement, while an unbounded mode could turn a prolonged sink outage into process OOM.

The writer enforces its limits at two layers. EventBuffer::write retains at most 64 KiB while formatting one event, so an oversized field cannot allocate an unbounded temporary before enqueue. The queue then admits a record only when both the configured record limit and aggregate payload budget permit it; the payload budget is tracked incrementally on push and pop rather than estimated from record count.

All normal severity levels share the same order-preserving queue. A separate priority queue for warnings and errors is omitted because it introduces cross-queue ordering ambiguity without guaranteeing durable delivery.

Configuration and bootstrap decision¶

logging.queue_capacity, logging.queue_capacity_bytes, and logging.file are restart-required. They are read once before the subscriber is installed by a best-effort peek at the configuration file (the same pass that sizes the Tokio runtime from server.workers).

The full configuration load happens afterward and logs through the pipeline configured by the peek, avoiding a split bootstrap where early startup logs use one writer and later logs use another. When the peek cannot read or parse the file, the pipeline starts with the compiled-in defaults and the main configuration load reports the error.

The loader validates these fields alongside the rest of the configuration:

• logging.queue_capacity: 1 to 1,000,000 records.
• logging.queue_capacity_bytes: 64 KiB (one maximum-size record) to 1 GiB.
• logging.file filename template: must contain {sequence}, no path separator, and no placeholder other than {date}, {time}, {pid}, and {sequence}.

Out-of-range queue bounds are refused at load so the pipeline never panics on startup assertions. The shutdown budget stays fixed.

One-off commands (config, metadata, auth, mcp-serve, and --migrate-config-file) always log to stderr with the compiled-in defaults and never open the file sink. Because they may inspect a configuration other than the active server config, they must not rotate or prune the server's log directory as a side effect.

Environment-only bootstrap settings are not provided, avoiding a parallel configuration surface that would require separate validation and documentation.

Test strategy¶

Testing is divided into deterministic correctness tests and repeatable performance experiments. Performance numbers alone cannot establish safety.

1. Deterministic writer tests¶

Use a controllable Write implementation rather than a real terminal or container runtime. It must support three modes:

• fast: consumes bytes immediately;
• throttled: permits a known number of bytes per second;
• stalled: blocks until the test releases a gate.

Tests must prove:

1. Emitting an event returns within a short bounded deadline while the sink is stalled.
2. More events than queue capacity does not increase resident queue storage beyond the configured bound.
3. Saturation increments the dropped-line count by the number rejected.
4. After the sink resumes, accepted records drain in FIFO order.
5. A disconnected or failing writer does not panic an emitting task.
6. Dropping ordinary cloned writer handles does not terminate the worker.
7. Dropping the owning guard performs the documented finite flush behavior.

Avoid timing-only assertions where a channel or gate can establish the same property deterministically. Deadlock checks need a timeout so a regression fails the test suite instead of hanging it.

2. Subscriber integration tests¶

Install the real formatting layer with the controlled writer and verify:

• HTTP mode selects stdout and stdio mode selects stderr;
• formatted records retain level, target, message, and structured fields;
• newline escaping and secret-redaction expectations remain unchanged;
• the otel and non-otel subscriber compositions both enqueue formatted output;
• ordinary log events remain excluded from OTLP export;
• initialization keeps the worker guard alive for the full process lifecycle.

Tests that capture tracing events must continue to install the repository's permissive global default before constructing their capture subscriber, as required by tests/tracing_capture_support.

3. Process and shutdown tests¶

Run the actual binary with stdout or stderr connected to a pipe controlled by the test process.

• Do not consume the pipe until its kernel buffer fills, then verify /health and a representative proxy route still respond.
• Resume consumption and verify the process remains healthy.
• Send SIGTERM with a writable sink and verify accepted records flush.
• Send SIGTERM with a stalled sink and verify exit occurs within the finite shutdown deadline.
• Verify MCP/ACP protocol messages on stdout are never mixed with diagnostics.

These tests protect the wiring that unit tests cannot see, especially guard ownership and destination selection.

4. End-to-end load experiment¶

Use a local mock LLM backend so provider latency and rate limits do not obscure Router behavior. Run release builds on an otherwise idle host and record the exact commit, CPU count, Tokio worker count, file-descriptor limit, command, payload, and concurrency.

Exercise both workloads:

• non-streaming: small fixed response, emphasizing request overhead;
• streaming: fixed-duration SSE response with a fixed chunk cadence, representing long-lived LLM connections.

For each workload, run these log destinations:

Run the same matrix before and after the change at several offered loads, including one below saturation and one above it. Use an open-loop load generator where possible so latency-induced coordinated omission does not make the slower system appear healthy.

Collect at least:

• achieved requests per second;
• p50, p95, and p99 latency;
• streaming time to first byte and stream completion time;
• error and timeout counts;
• process CPU and resident memory;
• Tokio worker saturation where observable;
• open file descriptors and TCP connection states;
• emitted, accepted, written, and dropped log counts;
• graceful shutdown duration.

Warm the binary and connection pools before recording samples. Repeat every case enough times to report the median and dispersion rather than selecting a single favorable run.

Acceptance gates¶

The change is acceptable only when all of these gates pass:

Correctness gate¶

• Existing log-content, security, OTLP, and protocol-separation tests pass.
• Under a fast sink, every accepted test record is written in order.
• Dropped-line accounting matches induced saturation exactly in deterministic tests.

Stability gate¶

• A stalled sink does not make representative HTTP requests miss their test deadline because of logging.
• Queue memory remains bounded during a sustained event storm.
• Sink failure does not panic or terminate the Router.
• A sustained full queue does not cause runaway CPU, allocation growth, or a recursive internal error storm.
• Health endpoints remain responsive while the sink is stalled and the log queue is continuously saturated.
• Shutdown completes within its documented bound even when the sink remains stalled.

Performance gate¶

• With a fast sink, the asynchronous implementation introduces no material regression outside normal benchmark variation.
• With a throttled or stalled sink, request throughput and tail latency remain materially closer to the no-log-sink baseline than the synchronous version.
• Any claimed improvement is supported by the recorded load-test artifacts, not inferred solely from microbenchmarks.

No fixed percentage is specified before the baseline is measured. The first benchmark run establishes variance and a reviewable regression budget; the budget must be written into the implementation change before it is merged.

Implementation sequence¶

1. Add the controlled writer test fixture and record the synchronous baseline.
2. Add a reproducible end-to-end load script and save baseline commands and results.
3. Introduce the bounded non-blocking writer and process-lifetime guard.
4. Expose dropped-line accounting without logging through the saturated path.
5. Add lifecycle, stdio separation, and OTLP coexistence tests.
6. Repeat the identical load matrix and compare results.
7. Run the normal repository checks, then make ci-local on the integrated result before merge.

Risks and mitigations¶

Non-goals¶

• Guaranteeing process-log delivery across SIGKILL, kernel failure, or host loss.
• Using logging as the billing or usage source of truth.
• Changing log field names or log format in the first implementation.
• Solving unrelated outbound connection-pool or HTTP/2 performance questions.
• Claiming Router-wide performance gains without an end-to-end measurement.