Skip to content

Migrating guardrails from LiteLLM

This page maps a LiteLLM proxy guardrail (a Python CustomGuardrail class and its hooks) onto Continuum Router's guardrail providers. The full option table for each provider is in Guardrails.

Concept mapping

LiteLLM concept Continuum equivalent
A CustomGuardrail class and apply_guardrail The webhook provider: run the class as a small HTTP service and keep its logic, with no in-process plugin
Masking or transforming logic in an async_pre_call_hook The pii provider (default_action: mask, phase: sanitize, built-in scanners plus options.custom_entities), or webhook when the service returns transform spans
Blocking logic in an async_pre_call_hook custom_classifier (a general chat model driven by a policy prompt), self_hosted_classifier (guard-model templates), or webhook (an operator-run rules service)
async_moderation_hook Any inspect provider on stages: [input] (openai_moderation, a classifier). It runs before dispatch, not in parallel with the call
async_post_call_success_hook stages: [output] (output gating)
async_post_call_streaming_iterator_hook Streaming output gating is built in (no separate implementation)
An NER service (Microsoft Presidio) The pii provider's Presidio-compatible external.endpoint
mode: pre_call / during_call / post_call stages: [input, output]. phase (sanitize or inspect) orders providers within a stage; it is not a timing. There is no during-call input pass: every input provider runs once, before dispatch
default_on guardrails.enabled: true plus the provider's enabled (default true). Every enabled provider runs on every gated request; there is no per-request client opt-in
Per-key guardrails Per-model guardrails.routes and Hub tier policy
Allowlists and bypasses guardrails.bypass_api_keys (keys that skip guardrails entirely)

Continuum's verdict order (block/mask/transform), categories, failure behavior (on_error), and timeout budgets are documented in the Concepts and Providers sections of Guardrails.

Steps

  1. Classify what each hook does: a hook that rewrites text moves to pii or webhook; a hook that only judges moves to a classifier, or to webhook when the decision is code the operator already runs.
  2. For a pii hook, enumerate the target entities and map them onto the built-in scanners (email, ssn, kr_rrn, and others) and custom regular expressions (options.custom_entities). A Presidio NER service connects through external.endpoint.
  3. For a blocking hook, move the policy into a plain-language policy prompt on custom_classifier, use self_hosted_classifier when a guard-model template (granite_guardian, llama_guard, and others) fits, or wrap existing code in an HTTP service and connect it through webhook.
  4. Map mode onto stages: pre_call becomes [input], post_call becomes [output], and both together become [input, output].
  5. Move the litellm_config.yaml guardrail list onto guardrails.providers[] and set guardrails.enabled: true and any routes.
  6. Choose the mode. guardrails.mode defaults to monitor, which records verdicts but never blocks or masks. Run in monitor first to compare verdicts with the old hooks, then set mode: enforce.

Worked example

A LiteLLM hook that masked an employee ID (EMP-\d{6}), masked emails, and blocked resident registration numbers moves into this one block:

guardrails:
  enabled: true
  mode: enforce
  providers:
    - name: pii-redaction
      type: pii
      enabled: true
      phase: sanitize
      stages: [input, output]
      options:
        default_action: mask
        actions:
          email: mask
          ssn: block
          kr_rrn: block
          employee_id: mask
        placeholder_format: "[REDACTED:{TYPE}]"
        custom_entities:
          - name: employee_id
            pattern: '(?-u:\b)EMP-\d{6}(?-u:\b)'
            context_words: ["employee", "사번"]
            context_window: 64
      on_error: fail_open

How this behaves:

  • mode: enforce makes the verdicts take effect; under the default monitor mode nothing would be masked or blocked.
  • email and employee_id are masked as [REDACTED:{TYPE}] ({TYPE} becomes the upper-cased entity type); ssn and kr_rrn block the request.
  • phase: sanitize masks before any inspect provider runs, so a classifier added later never sees the raw values.
  • The built-in scanners run locally with no external dependency, and raw detected values are never logged.
  • employee_id is a custom regular-expression entity; context_words reduces false positives by checking adjacent words.

A second blocking policy (for example, blocking competitor mentions) is one more provider on the same list:

    - name: competitor-guard
      type: custom_classifier
      enabled: true
      endpoint: "http://127.0.0.1:11434/v1/chat/completions"
      stages: [input]
      options:
        model: "granite3.1-dense:8b"
        policy_prompt: |
          Mark content unsafe if it mentions a competitor's product by name.
        default_category: competitor_mention

What has no equivalent

  • An in-process Python plugin: Continuum has no extension point for code; the hook's logic moves into one of the provider types above. Code-only rules such as a customer-list, checksum, or company-dictionary check run as an out-of-process service behind webhook: the router posts {stage, model, texts} and reads {decision, category?, confidence?, reason?, spans?}, so the service need not emulate a chat model or a Presidio analyzer. A chat-shaped endpoint can still use custom_classifier.
  • A during_call input pass: every input provider runs once, before dispatch. There is no mid-stream input inspection.