Decision Models¶

A decision model answers typed questions about a single document in one forward pass and returns a probability distribution for each question without generating free-form text.

Continuum Router supports decision models in two ways:

• Client-facing API: Exposes decision inference directly at POST /v1/decisions (alias POST /v1/systemone).
• Internal consumers: Uses decision models across five router subsystems:
  1. Content screening (Guardrail provider)
  2. Request classification (Smart-routing classifier)
  3. Agent tool-call evaluation (Action safety)
  4. Answer-quality fallback (fallback.quality_check in Model Fallback)
  5. Post-response quality telemetry (usage.quality_judgment)

A decision model's reported confidence measures how concentrated the output distribution is, not its empirical accuracy on your workload. Always calibrate thresholds against labeled traffic from your own deployment before enabling enforcement. Every shipped default in Continuum Router is derived from a reproducible calibration run committed to this repository, and any consumer whose measured accuracy does not beat a constant-answer baseline defaults to monitor-only or shadow mode.

Supported models¶

The router implements the System One protocol across four provider profiles (generic, typesafe, foundry, and llama_cpp; see Provider profiles and experimental media) and includes built-in catalog metadata for both hosted services and self-hosted open-weights checkpoints:

Wire protocol¶

POST /v1/decisions (or POST /v1/systemone) accepts a model, a state (string, JSON object, JSON array, or null on supported llama_cpp families), and a questions map. Each question uses one of four types:

• noul: Binary probability (0.0 to 1.0) that a statement is true, with optional criteria: {"true": ..., "false": ...}.
• choice: Categorical probability distribution over 1 to 255 named options (1 to 52 for openjev), returning the argmax choice, probabilities, and optional confidence.
• score: Ordinal probability distribution over ordered levels (2 to 10 levels on generic, foundry, and llama_cpp; 1 to 10 levels on typesafe), returning the probability-weighted score index, optional legend, probabilities, and optional confidence.
• box: Normalized bounding box (box: {top, bottom, left, right} in 0.0 to 1.0 plus optional confidence), supported on protocol: typesafe when typesafe_images: true and model is jev-1.14.0 (jev-preview).

{
  "model": "jev-latest",
  "state": {"request": "Write a Rust function that parses a CSV file", "message_count": 1},
  "questions": {
    "domain": {
      "type": "choice",
      "instructions": "What domain does `request` belong to?",
      "criteria": {"general": "...", "code": "...", "reasoning": "..."}
    },
    "complexity": {
      "type": "score",
      "instructions": "How hard is `request`?",
      "criteria": ["trivial", "simple", "moderate", "complex", "expert"]
    }
  }
}

The response echoes the question IDs, the resolved model, and usage (input_tokens and output_tokens).

Internal consumers¶

Continuum Router uses decision models across five internal consumers:

See also: Guardrails, System One Classifier, Claude Code Integration, and Model Fallback.

Usage accounting for decision calls¶

Every decision call that reaches an upstream server (including retries) is recorded as a usage event linked to the parent request with a consumer-specific purpose:

• guardrail for the System One guardrail provider
• classifier for the smart-routing classifier
• action_safety for the action-safety evaluator
• quality_judgment for both usage.quality_judgment and fallback.quality_check
• client for direct client requests to POST /v1/decisions (see "Router-initiated calls" in Architecture)

The target setting count_toward_budgets controls only the counted label on smart_routing_system_one_classifier_input_tokens_total.

Usage quality judgment (usage.quality_judgment)¶

usage.quality_judgment evaluates served chat replies asynchronously and attaches the resulting probabilities (in basis points, alongside the resolved judge_model) to the request's control-plane usage record. This allows Continuum Hub to compare response quality across models, API keys, and experiment arms without adding latency to the client path.

Asynchronous execution and late vs. skipped outcomes¶

• Non-blocking background task: Evaluation runs on a background task after the reply is served and never blocks usage accounting or batch uploads.
• late judgments: If a judgment finishes after its usage record's upload batch has already been sealed, the judgment is dropped and counted as late. Because slower judgments are more likely to miss the batch cutoff, a high late rate skews attached scores toward faster requests. To reduce late drops, increase control_plane.usage_push_interval_secs or use a low-latency self-hosted decision model.
• Streaming sampling: Streamed replies are sampled immediately after the input guardrail and evaluated as soon as the stream-end usage record is emitted. Only the head and tail of the client-visible text (bounded by max_reply_bytes, excluding reasoning tokens) are retained in memory.
• Unified stream label: Every outcome in usage_quality_judgments_total carries stream="true" or stream="false".

Route-by-route extraction and skip rules¶

See the usage section in the Complete Configuration Example for configuration syntax.

Call sharing with fallback.quality_check¶

For non-streaming /v1/chat/completions requests, fallback.quality_check and usage.quality_judgment can share a single upstream decision call when all of the following hold:

1. Both consumers are enabled and reference the same target configuration.
2. Both configure identical max_request_bytes and max_reply_bytes limits.
3. The request does not send tool results to usage.quality_judgment (include_tool_results is false or the turn has no tool results).

When a call is shared:

• The quality check includes language_match (when applicable) alongside answered, refusal, and truncated in the same forward pass. A missing or malformed language_match answer never fails the fallback check.
• If the initial reply passes the quality check, usage.quality_judgment reuses the returned probabilities immediately without consuming a max_in_flight slot or risking a late drop.
• If the shared call omits an answer for a required usage question, usage.quality_judgment issues a follow-up call for only the missing question(s) and merges the results (dropping the judgment as invalid if the follow-up resolves a different judge_model).
• If fallback.quality_check triggers a fallback hop to another model, usage.quality_judgment records skipped and makes no extra call.
• If fallback.quality_check did not run (for example, no fallback hop was available or its budget was exhausted), usage.quality_judgment performs its own normal background call.
• Shared calls increment systemone_requests_total{consumer="fallback_quality"}, while usage_quality_judgments_total{outcome} still records one outcome per sampled request.

Serving decisions to clients¶

Clients can call POST /v1/decisions (alias POST /v1/systemone) directly under the router's standard API key authentication, scopes, rate limits, budgets, and fallback chains:

• Native routing: Models listed under decisions.targets[] are forwarded directly to the configured System One server.
• Chat-model emulation: Any model not listed in decisions.targets[] is emulated on a chat backend using a forced emit_decision tool call. Emulated responses return one-hot probabilities, omit confidence, and set emulated: true so callers can distinguish emulated answers from native probability distributions.

The public endpoint acts purely as a proxy or emulator and does not apply router thresholds. See API Reference for request and response schemas.

Egress and operational caveats¶

• Data egress: Every decision call sends a state object constructed from the request:
  • Guardrail: Last user turn and a bounded conversation history (history_turns, max_turn_bytes).
  • Smart-routing classifier: Last user message and a 200-character system prompt excerpt. On /v1/chat/completions, the classifier state (including shadow mode) is built after the input guardrail stage, so blocked requests are never sent and PII-masked prompts are sent in their masked form.
  • Action safety: Proposed tool name, full arguments, and the user's latest instruction.
  • Answer-quality fallback & Usage quality judgment: Head-and-tail-capped user message and assistant reply (plus current-turn tool results for usage.quality_judgment only when include_tool_results: true).
  • Deployments that prohibit sending prompt or reply excerpts to third parties should point all consumers at self-hosted endpoints rather than hosted services such as TypeSafe Jev.
• Adversarial robustness: Decision models are fast screening and telemetry signals, not standalone adversarial defenses. On the reference guardrail corpus, uncalibrated Laya checkpoints score at or below the constant-answer baseline on most categories at a 0.5 cutoff and frequently misclassify non-Latin text with high confidence. Use decision models in monitor mode or alongside dedicated safety models (such as Llama Guard or Qwen3Guard) on enforce paths.
• Script sensitivity: The Laya english checkpoint does not support non-Latin scripts. Stock laya-serve routes non-Latin inputs to the multilingual checkpoint, which has lower accuracy on English and Korean routing tasks. Always evaluate English, Korean (ko), and non_latin slices separately before enabling enforcement.

When to use a decision model¶

Provider profiles and experimental media¶

Each System One target block selects an explicit wire profile via protocol (generic, typesafe, foundry, or llama_cpp), independent of the model alias:

• generic (default): Text-only System One contract.
• typesafe: Enables TypeSafe's structured criteria schemas and opt-in feature headers (typesafe_images, typesafe_long_context).
• foundry: Uses Azure AI Foundry's provider path (/providers/microsoft/v1/systemone) and deployment naming.
• llama_cpp: Enables llama-server multimodal data URLs and checkpoint-family constraints (model_family).

decisions:
  enabled: true
  targets:
    - name: jev
      protocol: typesafe
      endpoint: https://api.typesafe.ai/v1/systemone
      api_key_env: TYPESAFE_API_KEY
      model: jev-1.14.0
      typesafe_images: false         # opt in for JSON assets and Box
      typesafe_long_context: false   # independent opt-in
    - name: microsoft-decision
      protocol: foundry
      endpoint: https://RESOURCE.services.ai.azure.com
      auth: api_key                 # auto also selects api-key for Foundry
      api_key_env: AZURE_FOUNDRY_API_KEY
      model: support-decisions      # deployment name, not a catalog alias
    - name: local-decision
      protocol: llama_cpp
      endpoint: http://127.0.0.1:8080/v1/systemone
      model: local-decision
      model_family: generic         # openjev enforces its 52-choice bound

Target configuration reference (SystemOneTargetConfig)¶

The same target schema is shared across decisions.targets[], guardrails.providers[].systemone, smart_routing.classifier.system_one, safeguards.action_safety.systemone, fallback.quality_check.target, and usage.quality_judgment.target:

Protocol profile comparison¶

Profile details and wire examples¶

Foundry authentication and transport¶

• Endpoint resolution: Resource roots, /providers/microsoft, /providers/microsoft/v1, and /providers/microsoft/v1/systemone all normalize to /providers/microsoft/v1/systemone. Foundry requires an inline endpoint (backend: references are rejected). Remote endpoints must use HTTPS; HTTP is permitted only for loopback addresses (localhost, 127.0.0.0/8, ::1).
• Token lifecycle: auth: entra reads an externally provisioned Entra token from api_key_env at client build time and sends it as a Bearer token. It does not fetch or refresh tokens automatically; HTTP 401/403 responses are treated as terminal errors without retry.

Payload limits and TypeSafe box / assets¶

• Structural bounds: Individual string fields are capped at 8 KiB, JSON nesting at 64 levels, and score questions at 10 levels. Public ingress accepts up to 2 MiB of uncompressed JSON (multipart and compressed request bodies are rejected with HTTP 415).
• TypeSafe images and box: Setting typesafe_images: true enables top-level assets and sends TypeSafe-Danger-Zone-Images: true. box questions additionally require jev-1.14.0 or jev-preview. Client-supplied headers are never forwarded to enable disabled features.

{
  "model": "jev",
  "state": {"photo": "asset-photo-1"},
  "assets": {
    "asset-photo-1": {
      "type": "image",
      "format": "png",
      "data_base64": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="
    }
  },
  "questions": {
    "location": {
      "type": "box",
      "instructions": "Locate the red rectangle in photo."
    }
  }
}

A box answer returns normalized coordinates (0.0 to 1.0) and optional confidence:

{
  "model": "jev-1.14.0",
  "answers": {
    "location": {
      "type": "box",
      "box": {"top": 0.12, "bottom": 0.64, "left": 0.2, "right": 0.85},
      "confidence": 0.91
    }
  },
  "usage": {"input_tokens": 310, "output_tokens": 12}
}

llama.cpp multimodal inputs¶

protocol: llama_cpp accepts inline files or images data URLs (or multimodal message state with image/audio data URLs) up to max_media_bytes (default 4 MiB, max 16 MiB). External http(s) media URLs, local file paths, and native video transports are rejected.

{
  "model": "local-decision",
  "state": "Does this receipt show a total above $50?",
  "images": [
    "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII="
  ],
  "questions": {
    "over_fifty": {"type": "noul", "instructions": "Is the receipt total above $50?"}
  }
}

Model discovery, fallback, and accounting¶

• Model discovery (/v1/models): The model fetcher normalizes both TypeSafe's GET /v1/models format and llama.cpp's architecture.output_modalities array. Models advertising "decisions" receive the decisions capability; models advertising both "text" and "decisions" (such as Granite Switch 4.1) receive both chat and decisions.
• Fallback and emulation gates: Every fallback hop validates profile compatibility. Requests using media, state: null, or box questions cannot fall back to incompatible profiles or text-only chat emulation. Decision-only catalog models require a configured native target and are never emulated via chat completions.
• Token accounting: Both input_tokens and output_tokens are recorded in usage statistics even when a provider prices output tokens at $0. Missing confidence fields remain absent rather than synthesized.

Smart-routing classifier rollout¶

Smart routing uses the same target profiles under smart_routing.classifier.system_one. To evaluate a decision model safely before switching routing authority, keep method: rule (or llm / hybrid) and enable shadow: true:

# TypeSafe Jev (pin a specific calibrated version)
smart_routing:
  classifier:
    method: rule
    system_one:
      shadow: true
      protocol: typesafe
      endpoint: https://api.typesafe.ai/v1/systemone
      api_key_env: TYPESAFE_API_KEY
      model: jev-1.14.0

# Azure AI Foundry (model is your Foundry deployment name)
smart_routing:
  classifier:
    method: rule
    system_one:
      shadow: true
      protocol: foundry
      endpoint: https://RESOURCE.services.ai.azure.com
      auth: api_key
      api_key_env: AZURE_FOUNDRY_API_KEY
      model: support-decisions

# Self-hosted llama.cpp
smart_routing:
  classifier:
    method: rule
    system_one:
      shadow: true
      protocol: llama_cpp
      endpoint: http://127.0.0.1:8080/v1/systemone
      model: kev-local

Key rollout behaviors:

• Shadow evaluation: Shadow mode leaves routing decisions unchanged while recording agreement (smart_routing_shadow_agreement_total{field, agree}), latency, and token usage (purpose: classifier).
• Deterministic fallback: Under method: system_one, any call that fails, times out, omits confidence, or returns min(confidence) < min_confidence falls back to the rule classifier. Under method: hybrid_system_one, low-confidence answers fall back to the configured LLM classifier first, and then to the rule classifier.
• Candidate pool filtering: Decision-only models (decisions without chat/text) are automatically excluded from generation candidate pools and reported under excluded_incompatible_models on GET /admin/smart-routing/status.
• Cache invalidation: Classifier cache keys incorporate the bounded prompt state, target profile, model name, question battery, and backend topology generation (cache_target_identity), invalidating cached entries automatically on config or credential changes.

Calibrating a decision model on your own traffic¶

Because confidence is an uncalibrated concentration statistic, you should measure accuracy on your own labeled dataset before switching any consumer from monitor/shadow/advisory to enforcement.

1. Start a decision endpoint¶

By default, make calibrate-systemone launches the self-hosted CPU laya-serve container defined in deploy/docker-compose.systemone.yml on 127.0.0.1:8000 with all three Laya checkpoints preloaded.

To benchmark a single pinned Laya checkpoint (bypassing laya-serve's automatic script router) or a local Kev server, start the server manually:

# Pinned single Laya checkpoint
uv run --no-project --with 'laya-serve[inference]==0.1.1' \
  python scripts/systemone/laya_pinned_serve.py --checkpoint english --device cpu --port 8011

# Kev (uses MLX automatically on Apple silicon)
git clone https://github.com/jaredpalmer/kev && cd kev
uv sync --extra serve && uv run python -m kev.serve --run jaredpalmer/kev-0.8b --port 8008

2. Run the calibration harness¶

# Calibrate all three consumers against the default compose service (CPU)
make calibrate-systemone

# Run only the routing corpus (200 rows)
SYSTEMONE_CONSUMERS=routing make calibrate-systemone

# Run against a custom endpoint
SYSTEMONE_ENDPOINT=http://127.0.0.1:8008/v1/systemone SYSTEMONE_RUN_LABEL=kev-0.8b \
SYSTEMONE_MODEL=kev-latest SYSTEMONE_CHECKPOINT=jaredpalmer/kev-0.8b \
SYSTEMONE_CHECKPOINT_REVISION=54f4f8777356cd5bbbb6c6919c657f26e6f2f6d8 \
  make calibrate-systemone

make calibrate-systemone runs scripts/systemone/calibrate-compose.sh, executing the consumer calibration tests in tests/systemone_calibration_test.rs and writing raw results and derived thresholds to tests/fixtures/systemone/calibration/<date>/<label>/ (or SYSTEMONE_REPORT_DIR). Each row is replayed through the consumer's exact state builder and question battery. You can tune concurrency, timeout, and row count with SYSTEMONE_CONCURRENCY (default 1), SYSTEMONE_TIMEOUT_SECS (default 120), and SYSTEMONE_LIMIT.

To run a minimal live connectivity check without the full corpus, use the ignored smoke test:

SYSTEMONE_SMOKE_ENDPOINT=http://127.0.0.1:8000/v1/systemone \
SYSTEMONE_SMOKE_MODEL=laya-english \
SYSTEMONE_SMOKE_PROFILE=generic \
  cargo test --test systemone live_smoke_test::configured_native_endpoint_answers_a_text_decision -- --ignored --exact

3. Threshold derivation rules¶

tests/systemone_support/derive.rs sweeps a fixed threshold grid for each consumer and selects conservative thresholds where minimum evidence requirements are met (retaining vendor defaults otherwise):

• Guardrail category_thresholds.<c> (block): Smallest threshold in 0.50..=0.95 (step 0.05) achieving at least 0.90 block precision with at least 3 predicted blocks and 5 labeled positives.
• Guardrail review_threshold (flag): Largest threshold in 0.05..=0.50 maintaining at least 0.90 row-level recall across unsafe rows.
• Classifier min_confidence: Threshold in 0.00..=1.00 maximizing combined domain and complexity accuracy for the system_one -> rule fallback chain (ties broken toward the higher threshold). If no threshold beats the standalone rule classifier, no value is derived.
• Action safety: Derives thresholds only if action accuracy beats the constant-answer baseline. deny_threshold is the smallest value in 0.30..=0.95 with at least 0.90 deny precision on at least 3 denies; allow_threshold and min_confidence maximize allows with zero false-allow block labels, at least 0.95 allow precision, and at least 5 allows.

Shipped defaults¶

The shipped defaults are derived from the all slice of the reference run 2026-09-24/laya-serve-compose-cpu (stock laya-serve 0.1.1 on CPU with convaiinnovations/laya@55cf4c4ebb4e). tests/systemone_calibration_test.rs (shipped_defaults_match_the_reference_run_and_the_docs) verifies that the library constants and this table match the committed derivation.

These values map to guardrails.providers[].options.review_threshold, guardrails.providers[].category_thresholds, smart_routing.classifier.system_one.min_confidence, and safeguards.action_safety.systemone.{allow_threshold, deny_threshold, min_confidence}. Explicit values in config.yaml always take precedence.

Monitor-only findings¶

On the reference laya-serve-compose-cpu run, several consumers or language slices do not beat a constant-answer baseline and therefore default to non-enforcing modes:

• Guardrail (mode: monitor on all slices): At derived defaults, 70.8% of clean rows are flagged or blocked (all slice, block precision 0.652), and over 92% of clean ko and non_latin rows are flagged. Keep mode: monitor and use category probabilities as observability signals rather than standalone block gates.
• Smart-routing classifier (shadow: true recommended for Korean/non-Latin traffic): On the English (en) and combined (all) slices, the system_one + rule chain at min_confidence: 0.15 outperforms the standalone rule classifier (0.628 vs. 0.544 on en; 0.510 vs. 0.475 on all). On ko (0.405 vs. 0.487) and non_latin (0.418 vs. 0.428), Laya does not beat the rule classifier; Korean-heavy deployments should retain method: rule with shadow: true unless using a model such as kev-0.8b that outperforms the rule baseline on Korean (0.639 chain accuracy).
• Action safety (mode: advisory): On tool_calls.jsonl, action accuracy (0.319) is below the constant baseline (0.440). Keep mode: advisory.
• Unmeasured targets and consumers: Hosted TypeSafe Jev, Microsoft-Decision-1, kev-4b, and fallback.quality_check (default threshold 0.8 in mode: shadow) were not calibrated in the committed offline run. Always calibrate against your target endpoint and pin explicit model versions (such as jev-1.14.0 rather than jev-latest) before enforcing.

Results (2026-09-24)¶

The summary tables below are generated from the committed calibration fixtures under tests/fixtures/systemone/calibration/2026-09-24/ across three corpora:

• guardrail.jsonl: 244 rows (223 replayed live; 111 English, 66 Korean, 106 non-Latin)
• routing.jsonl: 200 rows (90 English, 79 Korean, 104 non-Latin)
• tool_calls.jsonl: 207 rows (English)

To regenerate the full per-question Brier, ECE, AUROC, and slice breakdown tables from the committed JSON files, run python3 scripts/systemone/results_tables.py 2026-09-24.

Evaluated endpoints¶

End-to-end latency and token usage by consumer¶

Routing shadow comparison¶

Comparison of standalone System One accuracy, standalone rule classifier accuracy, classifier agreement (smart_routing_shadow_agreement_total), and combined system_one -> rule chain accuracy across language slices: