AppProxy Worker Mode¶

This document specifies how Continuum Router operates as a Backend.AI AppProxy inference worker — a data-plane node that an AppProxy coordinator controls, aggregating many LLM serving containers behind a single OpenAI-compatible address.

It is the canonical reference for the worker mode, written to be precise enough that any engineer (or agent) can audit or extend each component without re-deriving the protocol.

The worker ships in two frontend modes:

• the legacy wildcard/port worker (appproxy v3, §§1–8): one circuit per frontend slot, Host-based ingress, per-circuit JWTs, model names auto-discovered from replicas;
• the ROUTER worker (§9): a single binding address for the whole cluster, model-name routing, and API-key-based model visibility.

This document specifies the router (worker) side. The ROUTER mode's companion on the Backend.AI side is BEP-1053: ROUTER Frontend Mode, which specifies the AppProxy coordinator and Manager changes: the FrontendMode.ROUTER value, slot-skipping circuits, the /v2/routers/* management APIs, desired-state persistence, and the key–model–service mapping surface. Where this document references coordinator or Manager behaviour, BEP-1053 is the authority.

1. Motivation¶

Backend.AI's AppProxy fronts model services with a transparent L4/L7 proxy worker: one circuit maps to one frontend slot (a port or a wildcard subdomain) and forwards bytes to the serving container(s) behind it, load balancing replicas by traffic_ratio.

Continuum Router can take over that inference data plane and add L7, LLM-aware behaviour on top:

• model-name routing and cross-endpoint aggregation behind one /v1 surface;
• protocol translation (OpenAI ↔ Anthropic ↔ Gemini), smart routing, prefix/KV-cache-aware routing, disaggregated prefill/decode;
• fallback chains, retries, response caching, Files API.

In worker mode, an AppProxy coordinator drives Continuum Router's backend set at runtime exactly the way it drives a stock worker (registration, heartbeat, circuit assignment), while Continuum Router realises each circuit as an LLM-aware, health-checked, weighted backend pool.

The slot-based shape (one frontend address per deployment) also leaves a gap at the cluster level, which the ROUTER mode (§9) exists to close:

• No single surface. Every deployment has a different base URL. Clients cannot hold one address and pick a model by name, the way every OpenAI-compatible SDK expects.
• No model abstraction. Users address a deployment, not a model. There is no first-class way to publish one model name backed by several deployment endpoints (for A/B testing two revisions, or for balancing one model across resource groups).
• No key governance. A deployment access token grants access to one deployment; there is no API-key surface that scopes which models a consumer may see and use across the cluster.

2. Background: AppProxy architecture¶

AppProxy has three parts (Backend.AI src/ai/backend/appproxy/):

• Coordinator — the control plane. An aiohttp REST server backed by PostgreSQL (the source of truth for workers, circuits, endpoints, tokens). It schedules circuits onto workers and pushes routing changes out.
• Worker — the data plane. Registers with the coordinator, heartbeats, and proxies traffic for the circuits assigned to it.
• Common — shared types, the event bus, and config.

Key entities¶

ROUTER mode adds three coordinator/Manager-side entities (specified in BEP-1053, consumed by the worker in §9):

Frontend modes¶

Transport (how coordinator and worker communicate)¶

There are three distinct channels:

1. Worker → Coordinator: HTTP REST. Registration, heartbeat, deregistration, and the initial circuit pull. Authenticated with a shared X-BackendAI-Token: <api_secret> header.
2. Coordinator → Worker: Redis Pub/Sub (legacy mode) — circuit create/route- update/remove broadcast on channel events_all-appproxy, with a worker ack on create.
3. Coordinator → Traefik: etcd (Traefik mode) — the coordinator writes Traefik dynamic config to etcd and Traefik proxies; the worker is not signalled per-circuit in this mode.

The mode is a coordinator-global setting (proxy_coordinator.enable_traefik). This distinction drives one of our design decisions (see §4 and §5.5).

ROUTER mode uses no additional channel: key–model–service mappings ride the same two worker channels, broadcast as Redis Pub/Sub events and served as a REST desired-state snapshot on the pull path (§9.9). There is no coordinator→router push connection; the coordinator never dials workers, so the worker keeps working behind NAT or a cluster ingress.

3. Conceptual mapping¶

The inference path maps almost 1:1 onto Continuum Router's existing model (this table is the legacy wildcard worker's mapping; the ROUTER-mode counterpart is in §9.2):

An AppProxy inference circuit is "one model's N replicas, weighted by traffic_ratio." That is exactly a Continuum Router backend group whose members share models = [<model>] and carry per-replica weight. The translation is therefore mechanical, and the shared data plane provides selection, health filtering, retry, and fallback.

4. Design decisions¶

Five decisions shape the integration:

1. Native module, not an external adapter. The integration lives inside Continuum Router behind Cargo features (§8). Circuits reach the data plane through the shared hot-reload config_sender channel.
2. Wildcard protocol compatibility. The legacy wildcard worker (§§5–8) speaks the wildcard inference-worker protocol as-is. Model names are obtained by auto-discovering each replica's /v1/models.
3. Wildcard ingress that honours the slot (legacy). The legacy worker registers a wildcard slot space and resolves each request to a circuit by HTTP Host (subdomain), with model-name aggregation available on a catch-all host. The coordinator-allocated slot is the ingress address, not a vestige (see §5.2).
4. Both transports. A pull-based reconcile baseline (works in any coordinator mode) plus a Redis Pub/Sub event overlay (legacy-mode support + low latency). Pull is also the backstop for missed events.
5. Both modes are supported. The ROUTER frontend mode (§9) is a departure from the slot model: it requires coordinator-side changes (BEP-1053) and replaces per-circuit JWTs with model API keys. Both modes coexist behind separate features and share one coordinator-client / event-codec / reconcile-apply core (§8).

5. Architecture¶

5.1 Component overview¶

                wildcard DNS: *.models.example.com  ──►  continuum-router host
                                  │
external client                   │   single socket (e.g. :443)
  POST https://ep-abc.models.example.com/v1/chat/completions
        │  Host: ep-abc.models.example.com
        ▼
┌────────────────────────────────────────────────────────────────┐
│ continuum-router  (one worker, frontend_mode = wildcard)         │
│                                                                  │
│  appproxy module (feature = "appproxy")                          │
│   ├── coordinator client (REST: register/heartbeat/pull)         │
│   ├── worker service (lifecycle loops) ── circuit registry       │
│   ├── reconcile (circuit → BackendConfig → config_sender) ──┐    │
│   ├── events (Redis Pub/Sub subscribe + ack)                │    │
│   └── ingress middleware (Host subdomain → model) ──┐        │    │
│                                                     ▼        ▼    │
│  existing pipeline:  model router → backend pool ◄── hot reload   │
│                      (health, retry, fallback)         │
└────────────────────────────────────────────────────────────────┘
        │              │
        ▼              ▼
   kernel1:port    kernel2:port    (LLM serving containers = backends)

The integration consists of the appproxy module and one ingress middleware. Circuit state becomes backend state through the existing hot-reload machinery; request routing reuses the existing model router.

5.2 Registration and the slot model¶

The router registers as a wildcard inference worker. "Single address" and "honour the slot" are not in conflict: the wildcard domain is the single address, and each circuit's subdomain is a virtual address into the same socket.

Registration advertises a slot space:

frontend_mode         = wildcard
wildcard_domain       = ".models.example.com"
wildcard_traffic_port = 443         # the router's /v1 socket
hostname              = <router host>
available_slots       = -1          # wildcard → unbounded; never runs out
accepted_traffics     = [inference]

The coordinator allocates one subdomain per inference circuit inside that domain, and Circuit.get_endpoint_url() produces https://ep-abc.models.example.com/. The Manager hands that endpoint URL to users. The operator configures wildcard DNS (*.models.example.com → router) once.

PORT mode (a port per circuit) would require the router to open and close listening sockets dynamically and is not supported (see §11). Wildcard + TLS is the norm for externally served inference.

5.3 Circuit → backend translation¶

Each inference circuit is translated to one BackendConfig per RouteInfo replica and applied through the existing runtime-mutation path:

for each circuit assigned to this authority:
    model = discover_model(circuit)            # from a replica's /v1/models, keyed by endpoint_id
    for each route in circuit.route_info:
        BackendConfig {
            name:         "appproxy-<circuit_id>-r<route_id>",
            backend_type: Generic,             # OpenAI-compatible; Vllm if known
            url:          "http://{route.kernel_host}:{route.kernel_port}",
            weight:       weight_from(route.traffic_ratio),
            models:       [model],             # what find_backends_for_model matches
            ..Default
        }

The apply step reuses the admin API's exact pattern (src/admin_config/backend_api.rs):

let _guard = config_modification_lock().write().await;   // serialise with admin API
let cfg    = state.current_config();                      // re-read under lock
let mut new_cfg = (*cfg).clone();
reconcile new_cfg.backends so that the set of appproxy-* backends
    equals the desired set derived from the current circuits;
state.config_sender.send(Arc::new(new_cfg));              // drives hot reload

HotReloadService then diffs old vs new, adds new backends, gracefully drains removed ones, syncs the health checker, and invalidates the model cache. No backend-pool code is touched. Backends owned by this module are namespaced with an appproxy- prefix so reconcile only ever adds/removes its own entries and never disturbs statically configured backends.

Two existing characteristics make this a good fit:

• Runtime config changes are in-memory only (never written to disk). The coordinator is the source of truth; the router re-syncs on restart via the initial pull. This is the desired behaviour, not a limitation.
• The typed backend pool is not hot-reloaded, which is irrelevant here: serving containers are generic OpenAI-compatible HTTP backends routed through the URL-based pool.

5.4 Ingress resolution (Host/subdomain → model)¶

An Axum middleware resolves the target circuit/model from the request:

1. Read the Host header; strip the configured wildcard_domain suffix to get the subdomain.
2. Look the subdomain up in the in-memory circuit registry (owned by the worker service, updated on reconcile/events) → circuit → canonical model.
3. Insert an IngressTarget { circuit_id, model } request extension.
4. For non-public inference circuits (open_to_public == false), verify the Authorization: Bearer <jwt> (HS256 with jwt_secret; the decoded id must equal the circuit id), matching AppProxy worker auth.

Handlers prefer the injected model over the body model field at the shared read sites. The request then enters select_backend_with_retry, where selection resolves the model name against each backend's models list.

Requests to the bare wildcard domain (or a configured aggregation host) skip subdomain scoping and use normal model-name routing across all circuits — this is the cross-endpoint aggregation surface.

Fallback participation (scoped fallback)¶

A registered circuit does participate in fallback.fallback_chains. When a request resolves to a circuit whose replicas are all down (its route_info is empty, so it has no live backend), the ingress middleware still pins the request to the circuit's canonical model and passes it to the normal pipeline. select_backend_with_retry then finds no backend for that model and FallbackService takes over, so a chain keyed on the circuit's model (e.g. vllm-real-poc → gpt-4o-mini) is reached — the "deployment went down, traffic goes to OpenAI" behaviour. Per-circuit auth (open_to_public, bearer token, allowed_client_ips) is enforced before this fall-through, so the fallback path is never an unauthenticated bypass.

The fall-through is scoped: it applies only to a registered circuit. A request to an unknown subdomain (no circuit in the registry) is still a 404 endpoint_not_found. An unknown subdomain is a circuit identifier, not a model name, so it does not enter the model-registry / fallback path.

5.5 Update transport¶

The worker keeps its circuit set current through two cooperating mechanisms:

• Pull reconcile (baseline, always on). After registering, the worker GETs /api/worker/{id}/circuits and reconciles; it then repeats on a timer (reconcile_interval). This alone is fully correct in Traefik mode (where the coordinator writes etcd and never signals workers) and is the backstop for any missed event.
• Redis Pub/Sub overlay (legacy mode + low latency). The worker subscribes to events_all-appproxy and applies create/route-update/remove deltas within ~1s, acking creates. This is required in legacy mode: the coordinator blocks up to 15 s on the worker's ack during circuit creation (initialize_legacy_circuit) and raises E10001 Proxy worker not responding on timeout. Route updates and removals are fire-and-forget.

Because the four circuit events are all broadcast (Pub/Sub), the worker needs only SUBSCRIBE (3 inbound) + PUBLISH (1 ack). Redis Streams / consumer-groups are not required for the circuit lifecycle.

6. Wire protocol reference¶

6.1 Coordinator REST API (worker scope)¶

Base URL = coordinator_url. Every request carries:

• X-BackendAI-Token: <api_secret>
• X-BackendAI-RequestID: <uuid4>

Registration request body (WorkerRequestModel), wildcard mode:

{
  "authority": "continuum-router-1",
  "frontend_mode": "wildcard",
  "protocol": "http",
  "hostname": "router.example.com",
  "tls_listen": false,
  "tls_advertised": true,
  "api_port": 8080,
  "accepted_traffics": ["inference"],
  "filtered_apps_only": false,
  "app_filters": [],
  "traefik_last_used_marker_path": null,
  "wildcard_domain": ".models.example.com",
  "wildcard_traffic_port": 443
}

The response includes the assigned id (worker UUID, cached for subsequent calls) and the computed slots.

6.2 Circuit and route data models¶

SerializableCircuit (the JSON shape returned by the REST snapshot and embedded in events):

RouteInfo:

Rust serde notes:

• Accept both kebab-case and snake_case input aliases (e.g. route-id and route_id); emit snake_case.
• extra = "ignore" semantics: tolerate unknown fields (#[serde(default)] / ignore unknown) so coordinator additions never break parsing.

6.3 Redis event envelope¶

All four circuit events are broadcast as a JSON object PUBLISHed to events_all-appproxy:

{
  "name": "<event_name>",
  "source": "<agent-id>",
  "args": "<base64(msgpack(args_tuple))>",
  "metadata": "{\"request_id\":null,\"user\":null}"
}

• args is base64 of a msgpack array. For these events the array elements are strings only — no msgpack ext types, no UUID/datetime/enum encoding at the msgpack layer (those are pre-encoded inside the inner JSON). A Rust impl needs only: JSON object → base64-decode args → msgpack array-of-strings → JSON for each element.
• metadata is a JSON string with exactly request_id and user (additional keys make the coordinator's parser raise). Emit {"request_id":null,"user":null} or echo the inbound request_id.
• source for worker-emitted events is "appproxy-worker". It is not used for routing; the worker filters inbound events on target_worker_authority.

Event payloads:

ROUTER mode adds five more event types in the same envelope; they are listed in §9.10.

Worked ack example (authority = "worker01", circuits_json = "[]"): msgpack(["worker01","[]"]) = 92 a8 worker01 a2 5b 5d → base64 kqh3b3JrZXIwMaJbXQ==, PUBLISHed to events_all-appproxy with name = appproxy_worker_circuit_added_event, source = appproxy-worker.

The Redis DB index for the event bus is the deployment's "stream" role DB and must be configured (redis_url / DB selector); confirm it against the coordinator's Redis profile.

7. Configuration¶

The optional router configuration section is gated by the appproxy feature (the ROUTER-mode section, appproxy_router, is specified in §9.11):

appproxy:
  enabled: true
  coordinator_url: "http://coordinator:10200"
  api_secret: "${APPPROXY_API_SECRET}"     # X-BackendAI-Token
  jwt_secret: "${APPPROXY_JWT_SECRET}"      # HS256 circuit/bearer verification
  redis_url: "redis://valkey:6379/4"        # event bus DB (stream role)
  authority: "continuum-router-1"
  hostname: "router.example.com"
  frontend_mode: "wildcard"
  wildcard_domain: ".models.example.com"
  aggregation_hosts: []                      # extra Hosts that skip subdomain scoping
  wildcard_traffic_port: 443
  tls_advertised: true
  heartbeat_period: "10s"
  reconcile_interval: "15s"
  events_enabled: true                       # Redis Pub/Sub overlay on/off

Secrets support ${ENV_VAR} interpolation, consistent with backends[].api_key.

aggregation_hosts is optional and empty by default. The bare wildcard apex (wildcard_domain without its leading dot, e.g. models.example.com) is always an aggregation surface implicitly; list any additional vanity or aggregation hostnames here. A request whose Host matches one of these (or the apex) skips per-circuit subdomain scoping and uses normal model-name routing across all circuits (§5.4).

8. Module layout¶

The worker is split into a shared core and a per-mode layer. The wildcard/port worker and ROUTER mode share one coordinator-client, event-codec, and reconcile-apply implementation.

src/appproxy/                 # top-level: re-exports the stable public surface
├── mod.rs                    # crate::appproxy::* re-exports; stable events:: facade
├── events.rs                 # public re-exports for appproxy::events::*
├── common/                   # feature = "appproxy-common" (shared core, no legacy deps)
│   ├── mod.rs                # module-level docs; reuse-base for ROUTER mode
│   ├── client.rs             # CoordinatorClient; REST register/heartbeat/pull; X-BackendAI-Token
│   ├── config.rs             # AppProxyWorkerConfig base
│   ├── events.rs             # envelope codec (base64+msgpack), EventHandler trait, reconnect subscriber
│   ├── reconcile.rs          # circuit → BackendConfig → config_sender (under the lock)
│   ├── status.rs             # /status management handler
│   └── types.rs              # SerializableCircuit, RouteInfo, enums (dual-alias serde)
├── legacy/                   # feature = "appproxy-legacy" (appproxy v3, requires common)
│   ├── mod.rs
│   ├── events.rs             # LegacyEventHandler: registry-driven circuit create/update/remove + ack
│   ├── ingress.rs            # Host subdomain → IngressTarget middleware; apply_ingress_model_override
│   ├── jwt.rs                # verify_circuit_token (HS256); JwtError
│   ├── registry.rs           # AppProxyRegistry: subdomain-keyed circuit store
│   └── worker.rs             # run_worker: register → pull → heartbeat → reconcile lifecycle
└── router/                   # feature = "appproxy-router" (ROUTER mode)
    ├── mod.rs
    ├── config.rs             # RouterWorkerConfig (no slot/JWT fields; redacting Debug)
    ├── client.rs             # RouterCoordinatorClient: ROUTER registration + router-config pull
    ├── events.rs             # ROUTER-mode Redis event payloads (model/key updated/removed + applied-ack)
    ├── keys.rs               # RouterKeyStore: hash auth, visibility, per-node rate limit, IP allow-list
    ├── overlay.rs            # ROUTER event overlay: subscriber glue + per-node applied-ack
    ├── reconcile.rs          # RouterReconcileState: circuits × publications → backends (+ key set)
    ├── types.rs              # ModelPublication, ModelApiKey, RouterRegisterRequest (wire types)
    └── worker.rs             # run_router_worker lifecycle; RouterKeyStore registration

The common/ tree is the documented reuse base for ROUTER mode: it builds standalone under appproxy-common and depends on no legacy-only symbol (jsonwebtoken, the subdomain registry, or the Host ingress layer). ROUTER mode plugs its own EventHandler into the same shared subscriber.

Wiring points:

• Cargo.toml: four features. appproxy-common = ["dep:redis", "dep:deadpool-redis", "dep:rmp-serde"] is the self-contained shared core (rmp-serde stays here because the msgpack envelope codec is shared); common reconciliation uses the neutral, always-compiled configuration-mutation lock, and ROUTER IP filtering uses the neutral allow-list helper, so neither path requires the Admin API feature. appproxy-legacy = ["appproxy-common", "dep:jsonwebtoken"] is the appproxy v3 worker (jsonwebtoken is legacy-only); appproxy-router = ["appproxy-common"] is the ROUTER frontend mode; and appproxy = ["appproxy-legacy"] is an accepted alias for the wildcard/port worker. None appear in full; official release binaries are built with appproxy-router enabled (the Release workflow), while appproxy-legacy is a source-build opt-in and ROUTER mode activates only via the appproxy_router config section. CI checks appproxy-common with default features disabled so an accidental Admin dependency cannot return unnoticed.
• src/lib.rs: pub mod appproxy;.
• src/core/config/models/config.rs: pub appproxy: Option<AppProxyWorkerConfig> and pub appproxy_router: Option<RouterWorkerConfig>.
• src/server/mod.rs::build_router: register the worker /status route (either feature) and insert the legacy ingress middleware just outside the rate-limit layer (so the resolved model is visible to the rate limiter).
• src/server/serve.rs: after the hot-reload block, spawn appproxy::run_worker(cfg, state.clone(), shutdown_rx.clone()) when cfg.appproxy.enabled (legacy), and appproxy::run_router_worker(...) when cfg.appproxy_router.enabled (ROUTER).
• src/http/middleware/auth.rs: the dynamic API-auth middleware consults the ROUTER key store when ROUTER mode is enabled (§9.7).
• src/models/handlers.rs: /v1/models visibility filtering against the published-model set (§9.7).

9. ROUTER worker (appproxy-router mode)¶

The ROUTER worker is the appproxy-router-feature counterpart of the legacy wildcard worker described in §§5–8. Where the legacy worker uses Host-based subdomain ingress and per-circuit JWT auth, the ROUTER worker advertises a single binding address with model-name routing and per-key model visibility: no subdomain dispatch, no per-circuit JWT, no slots.

This is a deliberate departure from the wildcard design: a wildcard subdomain per circuit reproduces the one-address-per-deployment shape, and forces DNS/TLS wildcards onto every deployment. In ROUTER mode the coordinator skips slot management entirely, Circuit.get_endpoint_url() returns the worker's single advertised base URL for every circuit, and what the Manager hands to users is the triple (base URL, model name, API key) instead of a per-deployment URL.

The coordinator/Manager side (the FrontendMode.ROUTER enum value, slot-skipping registration, the /v2/routers/* manager APIs, the router-config events, and the snapshot endpoint) is specified in BEP-1053; a ROUTER worker cannot register against a coordinator that predates it (§9.13).

9.1 What ROUTER mode provides¶

• A single OpenAI-compatible /v1 endpoint for the whole cluster; clients select a model by name in the request body, never by URL.
• A published model is a collection of one or more deployment endpoints, each with its replicas: two-level hierarchical routing (§9.2).
• API-key-based visibility: which models a client can list and invoke is determined by the API key in the request header (§9.7).
• High availability: multiple router workers may register under one authority behind a load balancer; the coordinator keeps them in sync (§9.5).
• The router's data plane applies health checking, weighted selection, retries, fallback chains, and protocol translation to Backend.AI replicas.

9.2 Concepts: publications, keys, and two-level routing¶

The central concept is a published model:

API key ──(visibility)──► model ──(level 1)──► deployment endpoint(s) ──(level 2)──► replica session

• A publication binds a model name (e.g. llama-4-chat), defined in Backend.AI Manager, to one or more deployment endpoints. Multiple endpoints behind one model name support A/B testing between revisions and load balancing across resource groups; each mapping carries a split ratio. A publication may also declare aliases, additional names that route identically, so one deployment can be exposed under several model names (e.g. gpt-4 and gpt-4-internal).
• Each deployment endpoint has one or more replica sessions (the circuit's route_info), weighted by traffic_ratio.
• ratio is a non-negative relative weight (it need not sum to 1.0 across a model's endpoints); ratio = 0 drains an endpoint: it stays mapped but receives no new traffic (§9.6).
• A model API key carries the set of model names it may see and use; primary names and aliases are gated independently. An empty allow-list means the key is unrestricted and may use any published model (§9.7).

The hierarchy maps onto the router's existing entities almost 1:1:

Conceptually selection is hierarchical: pick a model, then a deployment endpoint, then a replica. Internally the router realizes it as one weighted selection over a flattened replica pool, with the two levels composed into the weights:

1. Level 1 — model selection. The request's model field is looked up via find_backends_for_model, which returns every backend whose models list contains the name, i.e. the union of all replicas of all deployment endpoints mapped to that model. Because the reconcile populates each backend's models list with the publication's primary name and every alias (§9.6), a request naming any alias resolves to the same backend set with no separate alias-lookup step. The API key's allow-list is applied here: a name outside the key's set is rejected with 403 (and never appears in that key's /v1/models).
2. Level 2 — replica selection. The configured SelectionStrategy (weighted round-robin, least-latency, prefix-aware hash, …) picks one backend from the candidate set after the health checker excludes unhealthy replicas. Each candidate's weight composes endpoint_ratio × route.traffic_ratio (§9.6), so traffic splits correctly across endpoints and across replicas within each endpoint.

Flattening is deliberate, and is what makes multi-endpoint models cheap:

• Latency- and cache-aware strategies observe all replicas of a model at once, regardless of which deployment endpoint they belong to.
• If one mapped endpoint loses all replicas, its weight share simply disappears and traffic shifts to the surviving endpoints; A/B arms degrade gracefully with no control-plane round trip.
• Only when every replica of every mapped endpoint is gone does the model have no backend, at which point fallback.fallback_chains may take over (§9.7).

9.3 Component overview¶

   Backend.AI Manager ──(key–model–service mappings)──► AppProxy Coordinator
        │  user creates deployments,                        │ persists desired state
        │  defines models & API keys                        │ (PostgreSQL); schedules
        ▼                                                   │ circuits; broadcasts
   deployment endpoints (replica sessions)                  │ events (Redis Pub/Sub)
                                                            │
external client                                             │
  POST https://router.example.com/v1/chat/completions       │
        Authorization: Bearer <api-key>                     │
        {"model": "llama-4-chat", …}                        │
        │                                                   ▼
        ▼                              single socket (host:port)
┌─────────────────────────────────────────────────────────────────────────┐
│ continuum-router  (one worker, frontend_mode = router)                   │
│                                                                          │
│  appproxy::router module (feature = "appproxy-router")                   │
│   ├── coordinator client (REST: register/heartbeat/pull snapshots)       │
│   ├── worker service (lifecycle loops) ── RouterReconcileState           │
│   ├── reconcile (circuits × publications → BackendConfig → config_sender)│
│   ├── events overlay (Redis Pub/Sub subscribe + applied-ack)             │
│   └── RouterKeyStore (key hashes, visibility, per-node rate limits)      │
│                                                                        ▼ │
│  existing pipeline:  API-key gate → model router → backend pool ◄─ hot   │
│                      (health, retry, fallback)        reload   │
└─────────────────────────────────────────────────────────────────────────┘
        │              │              │
        ▼              ▼              ▼
   replica1:port  replica2:port  replica3:port   (LLM serving containers)

There is no ingress middleware and no Host-based scoping. Circuit and mapping state becomes backend and key state through the existing hot-reload machinery; request routing reuses the existing model router, and the API-key gate runs in the dynamic auth middleware (§9.7).

9.4 Lifecycle¶

1. Pin api_keys.mode = blocking before any traffic arrives. The worker forces the global API-key gate into blocking mode at startup. When the api_keys section is absent it is created empty; existing operator-configured keys are left untouched. This prevents unauthenticated requests from reaching the data plane during the brief window before the initial key set loads.
2. Register (PUT /api/worker) with the coordinator, carrying an ephemeral per-process node_id UUID. Authentication is X-BackendAI-Token, the same as the legacy worker. Transient failures retry with exponential backoff until shutdown. The response id is the worker_id used in all subsequent REST calls.
3. Initial pull and reconcile (pull_state): fetch the circuit snapshot and the router-config snapshot (model publications + API-key hashes) in one pass. RouterReconcileState joins circuits × publications into the desired appproxy-* backend set, applies the result through the hot-reload channel, and mirrors the key set into the in-memory RouterKeyStore the data-plane gate consults.
4. Heartbeat loop every heartbeat_period (default 10s): PATCH /api/worker/{id} carrying node_id. The coordinator uses node_id to refresh per-node liveness, which is distinct from the shared authority-level worker_id.
5. Revision-based reconcile loop every reconcile_interval (default 15s): re-pull with the last-applied known_revision. A 304 skips the apply; a 200 returns fresh data and a new revision. This is the backstop for any missed event.
6. Event overlay (using the same Redis Pub/Sub subscriber codec as the legacy worker): four inbound ROUTER-mode events (model-updated, model-removed, key-updated, key-removed) each funnel through the same reconcile mutex as the pull path, so events and pull reconciles never interleave inconsistently. Disabled with events_enabled: false; the pull loop is fully correct without it.
7. Graceful shutdown: the heartbeat and reconcile loops stop, the event subscriber task is aborted, and DELETE /api/worker/{id} deregisters the node.

9.5 Registration, HA, and per-node identity¶

The router registers as a ROUTER-mode inference worker: a single advertised address and no slot space.

frontend_mode     = router
hostname          = <advertised host>   # e.g. router.example.com (the LB, if any)
api_port          = 8080                # the router's socket (data plane + worker API)
traffic_port      = 443                 # advertised data-plane port; defaults to api_port
node_id           = <uuid4 per process> # per-node liveness identity
accepted_traffics = [inference]
port_range        = null                # no slot space
wildcard_domain   = null

The coordinator skips slot accounting entirely: a ROUTER worker's slot capacity is unbounded (like wildcard, so it is always a valid inference candidate) and circuits are created with neither port nor subdomain. How a deployment's circuit reaches this worker is a coordinator concern (BEP-1053): scaling groups select the coordinator via ScalingGroupProxyTarget, and pointing several scaling groups at the coordinator hosting this authority is what lets one router serve multiple scaling groups behind one model surface.

High availability and load balancing¶

Multiple router workers may serve the same logical router. They register under the same authority, typically behind an external L4 load balancer whose address is the advertised hostname. The coordinator delivers the same circuit set and the same key–model mappings to every node of the authority, so any node can serve any request. Synchronization across nodes is inherent, not a separate subsystem: all nodes share one coordinator-side worker identity, Pub/Sub events fan out to every node, and the authority-scoped pull snapshot reconciles each node to identical desired state (a late-joining or restarted node catches up via the full pull on registration). Distinct authorities may also be registered to one coordinator to partition deployments across independent routers.

Per-node identity and liveness. Each worker process generates an ephemeral node_id (UUID4) at startup and repeats it on registration and every heartbeat. The coordinator uses it to track per-node liveness (with TTL eviction) and to expose per-node health, while traffic routing away from a dead node stays the LB/VIP's responsibility. The node_id is also echoed in the config-applied ack (§9.10) so the coordinator's strict-revocation mode can count per-node acks. It is not persisted; a restart yields a new node_id and the old one expires.

Selection-strategy behaviour under HA. Because each node runs its own in-memory selection over the same backend set, deterministic strategies (e.g. prefix-aware hashing) independently pick the same replica on every node, so cache affinity is preserved with no cross-node coordination. Stateful strategies that depend on locally-observed signals (e.g. least-latency) make each node decide from its own view; this is acceptable but not globally coordinated. Likewise, per-key rate_limit is enforced per node (§9.7).

9.6 Reconcile: circuits × publications → backends¶

Each inference circuit assigned to this authority is joined with the model publications and translated to one BackendConfig per RouteInfo replica:

for each inference circuit with an endpoint_id:
    referencing = publications whose mappings name circuit.endpoint_id
    if referencing is empty: continue            # tracked, but contributes no backends
    models         = union of all_names(p) for p in referencing   # primary + aliases, deduped
    endpoint_ratio = max(sanitize(m.ratio) for every referencing mapping of this endpoint)
    for each route in circuit.route_info:
        weight = clamp(round(100 × endpoint_ratio × sanitize(route.traffic_ratio)), 1, 1000)
        if the composed ratio is 0: skip          # drained replica (no backend emitted)
        BackendConfig {
            name:         "appproxy-<circuit_id>-r<route_id | index>",
            backend_type: Vllm if runtime_variant == "vllm", else Generic,
            url:          "http://{route.kernel_host | localhost}:{route.kernel_port}",
            weight,
            models,                              # what find_backends_for_model matches
            ..Default
        }

Behavioural notes, in the order they matter operationally:

• Held circuits and orderless joins. A circuit whose endpoint is not (yet) mapped by any publication is still tracked but contributes no backends; it becomes routable the moment a publication names it. Conversely, a publication that references a circuit not assigned to this authority is held until the circuit arrives. The coordinator normally orders these, but the reconcile does not depend on ordering.
• Empty publications resolve to fallback. A published model whose mappings are empty (e.g. after its last endpoint was destroyed; the Manager prunes but does not auto-unpublish, BEP-1053) contributes no backends and behaves like an all-replicas-down model: it resolves to its fallback_chains if any (§9.7), otherwise model-unavailable. The name stays listed in /v1/models until an operator unpublishes it.
• Shared endpoints take the max ratio. When several publications map the same endpoint, the effective endpoint_ratio is the largest sanitised ratio among them. This keeps drain semantics (all-zero → every replica drained) while never letting one publication's drain starve another that still routes traffic through the same physical replica.
• Ratio sanitising. A non-finite or negative ratio/traffic_ratio from the coordinator is coerced to 0 (drain) rather than propagated, where it would otherwise become a max-weight traffic magnet or panic the float cast.
• Weight composition. The composed ratio is scaled by 100 and clamped into the backend pool's weight range (1–1000). A composed ratio of exactly 0 emits no backend at all (drain).

The apply step is identical to the legacy worker's (§5.3): under config_modification_lock, rebuild the appproxy-* backend set, and send the candidate config through config_sender. HotReloadService diffs, drains removed backends, syncs the health checker, and invalidates the model cache. The same pass also replaces the key set in the RouterKeyStore and the published-model list used for /v1/models visibility.

9.7 Request path: key gate, model routing, fallback¶

A request to the single endpoint flows through the router's normal pipeline:

1. API-key gate (dynamic auth middleware, src/http/middleware/auth.rs). The key is read from Authorization: Bearer <key> or the Anthropic-style x-api-key header. The worker holds only the SHA-256 hash of each key (§9.8), so it authenticates by hashing the presented bearer and looking it up in the RouterKeyStore. The store is worker-owned and separate from the legacy ApiKeyStore; reconciled keys take effect without touching the api_keys config section. Outcomes:

   • missing, unknown, or expired key → 401;
   • key carries allowed_client_ips and the peer IP matches no entry (individual IP or CIDR) → 403;
   • per-key rate_limit (requests/min, enforced per node) exceeded → 429;
   • otherwise an AuthContext carrying the key's model allow-list is attached for downstream gating.

2. Model visibility and routing. /v1/models lists exactly allowed_models ∩ currently-published-names; an empty allowed_models means unrestricted, which lists every published name. Primary names and aliases are gated independently. The body's model field selects the backend candidate set as described in §9.2 (level 1); a published name outside the key's allow-list is rejected with 403 (enforce_model_access), and the selection strategy picks a replica (level 2).

3. Fallback participation (scoped). A published model participates in fallback.fallback_chains: when all of its replicas are down, select_backend_with_retry finds no backend and FallbackService takes over, so a chain keyed on the model (e.g. llama-4-chat → gpt-4o-mini) is reached — the "deployment went down, traffic goes to OpenAI" behaviour. The key gate runs before this fall-through, so fallback is never an unauthenticated bypass. An unknown model name (no publication) is a plain 404 model_not_found and does not enter the fallback path.

Per-circuit JWT bearer tokens (the legacy worker's open_to_public == false auth) are not used in ROUTER mode; model API keys replace them as the data-plane credential, and a circuit's open_to_public flag is ignored. Source-IP allow-listing is likewise expressed per key (ModelApiKey.allowed_client_ips), not per circuit: with no Host-based circuit scoping there is no circuit identity at ingress time to attach a circuit-level CIDR list to.

9.8 Control plane: who defines what, who holds the keys¶

Backend.AI Manager adds a management surface (GraphQL/CLI/WebUI) for two objects, both scoped to a router authority (BEP-1053):

• Publication — primary name + aliases → [{endpoint_id, ratio}] with a control_mode: which deployment endpoints serve a published model, under which names, with which traffic split, and who owns those ratios (manual vs strategy_managed; the router routes identically either way and preserves the distinction for tooling).
• Model API key — key_id → {token_hash, allowed_models, expires_at, rate_limit, allowed_client_ips}: a credential a consumer presents to the router. The router stores only the hash.

The model API key is distinct from Backend.AI's deployment access token (the per-deployment JWT minted via the coordinator). WILDCARD/PORT frontends use deployment access tokens.

The flow of a mapping or key, end to end:

 user/admin                Manager                Coordinator                Router worker(s)
     │  define model /        │                       │                          │
     │  issue key             │                       │                          │
     ├──────────────────────► │  PUT /v2/routers/{authority}/…                   │
     │                        ├─────────────────────► │  1. persist (PostgreSQL) │
     │                        │                       │  2. broadcast event ───► │  apply mapping payload /
     │                        │                       │     (Redis Pub/Sub)      │  pull key hash (REST)
     │                        │                       │  ◄── ack (first node) ───┤  → hot reload
     │   key shown to user    │  ◄── 200 + ack status │  3. wait ≤15 s for ack   │
     │ ◄──────────────────────┤                       │                          │

Responsibilities, precisely:

• The Manager generates the keys; nobody stores the plaintext. Key material (an opaque sk-…-style token) is generated by the Manager, shown to the user exactly once, and persisted only as a SHA-256 hash (plus a masked display hint) alongside the owner, the allowed model set, and the expiry. The Manager is the single user-facing surface for issuing, listing (masked), rotating, and revoking keys; zero-downtime rotation is a second key_id issued before the first is revoked.
• The Coordinator persists the desired state and broadcasts it. The Manager never talks to a router directly; it calls the coordinator, which applies each mutation to PostgreSQL first (the desired state) and then announces it to the authority's workers as a Pub/Sub event (§9.10). The call is proactive-deploy: the coordinator waits up to 15 s for the first worker ack (the same pattern initialize_legacy_circuit uses), then returns. It never waits for every node, and on ack timeout the call still succeeds: the persisted state is authoritative and pull reconcile guarantees convergence. Coordinator-side persistence is required, not optional, because router runtime state is in-memory by design and must be replayable whenever a node (re-)registers.
• The Router holds key hashes in memory only (the RouterKeyStore; api_keys.persistence_file is unused in ROUTER mode) and enforces them on the data plane as described in §9.7.

No recoverable secret rides the event bus — or moves at all after issuance. Publication events carry their full payload (a publication contains no secrets), but key events are notifications only, (authority, key_id), and the worker fetches the key's hash over the authenticated REST snapshot pull (§9.9). The Redis bus is shared infrastructure; circuit events already transit it but carry kernel addresses, never credentials, and this design preserves that property. With hash-only custody the plaintext token never leaves the user after issuance: the two point-to-point HTTP hops (Manager → Coordinator, Coordinator → Router pull) carry only hashes, which must run over TLS or a trusted network and must never be logged. A leaked hash is not bearer-equivalent (the router hashes what the client presents), so a coordinator/Manager DB compromise leaks nothing directly usable.

Revocation is two-tier. An explicit key revoke (DELETE …/api-keys/{key_id}) propagates immediately via the id-only removal event, applied on receipt with no fetch; a node that misses the notification converges at its next pull, so the worst-case window on that node is reconcile_interval. An RBAC-driven implicit revoke (the owner loses access, so the Manager shrinks the model out of the key) is additionally bounded by the Manager's periodic reconcile interval. For immediate hard revocation, DELETE …?strict=true makes the coordinator wait for an ack from every live node (each ack carries its node_id, §9.10) and report any unconfirmed nodes; the LB/VIP is still what removes a dead node from traffic.

9.9 Update transport¶

The worker keeps its circuit set, publications, and keys synchronized through two mechanisms:

• Pull reconcile (baseline, always on). After registering, the worker GETs /api/worker/{id}/circuits and the ROUTER-mode desired-state snapshot (/api/worker/{id}/router-config), and reconciles by set-diff against its in-memory state. It repeats on a timer (reconcile_interval). This is the backstop for any missed event and what restores the full in-memory state after a restart. A single per-authority revision covers both publications and keys; the worker passes its last-applied value as GET /api/worker/{id}/router-config?known_revision={r}, and an unchanged revision yields a cheap 304 Not Modified (an explicit query param rather than an HTTP ETag, since the coordinator's REST layer has no conditional-GET middleware). On change the coordinator returns the combined full snapshot and the worker set-diffs the whole thing.
• Redis Pub/Sub overlay (low latency). The worker subscribes to events_all-appproxy and applies deltas within ~1 s. Circuit, publication, and key events share this bus (§§5.5, 9.10). Every inbound event carries the full state of one entity (or its deletion), so application is idempotent and last-writer-wins, exactly like appproxy_circuit_route_updated_event replacing a circuit's whole route table; there is no version chain to track. On the worker, every event funnels through the same reconcile mutex as the pull path, so events and pull reconciles never interleave inconsistently.

Because all of these events are broadcast (Pub/Sub), the worker needs only SUBSCRIBE + PUBLISH (acks). Redis Streams / consumer-groups are not required.

9.10 Wire protocol additions¶

Worker-scope REST (delta over §6.1)¶

Registration request body, ROUTER mode:

{
  "authority": "continuum-router-1",
  "frontend_mode": "router",
  "protocol": "http",
  "hostname": "router.example.com",
  "tls_listen": false,
  "tls_advertised": true,
  "api_port": 8080,
  "traffic_port": 443,
  "node_id": "b3f1c0de-…",
  "accepted_traffics": ["inference"],
  "filtered_apps_only": false,
  "app_filters": [],
  "port_range": null,
  "wildcard_domain": null,
  "wildcard_traffic_port": null,
  "traefik_last_used_marker_path": null
}

traffic_port is the advertised data-plane port (an LB may sit in front), defaulting to api_port when the config leaves it unset. node_id is the ephemeral per-process UUID (§9.5); the same value is sent on registration and every heartbeat. The slot fields are emitted as explicit null, matching the legacy registration body's convention. The response is {"id": <worker uuid>}, shared across all nodes of the authority.

router-config snapshot response (200):

{
  "revision": "42",
  "models": [
    {
      "model": "llama-4-chat",
      "aliases": ["llama-4"],
      "mappings": [
        {"endpoint_id": "3333…", "ratio": 0.7},
        {"endpoint_id": "4444…", "ratio": 0.3}
      ],
      "control_mode": "manual"
    }
  ],
  "api_keys": [
    {
      "key_id": "key-abc",
      "token_hash": "<sha256 hex>",
      "allowed_models": ["llama-4-chat"],
      "expires_at": "2026-12-31T23:59:59Z",
      "rate_limit": 600,
      "allowed_client_ips": ["10.0.0.0/8"]
    }
  ]
}

Serde rules match §6.2: snake_case output, kebab-case input aliases accepted (models also accepts publications; api_keys accepts keys/api-keys), unknown fields tolerated. revision is opaque to the worker: it accepts a JSON string or number, stores it as a string, and only ever compares and echoes it. Per key, allowed_models empty means unrestricted; expires_at, rate_limit (requests/min, per node), and allowed_client_ips (IPs or CIDRs) are optional. control_mode ∈ manual (default) / strategy_managed; ratio defaults to 1.0.

Manager-scope REST (coordinator side, BEP-1053)¶

The Manager manages publications and keys through the coordinator, which persists and broadcasts them (§9.8). The router never calls these; they are listed for context. All routes are authenticated with the shared X-BackendAI-Token secret:

Events (delta over §6.3)¶

Five event types ride the same events_all-appproxy envelope:

9.11 Configuration and mode selection¶

The ROUTER worker is configured by a dedicated appproxy_router section, gated by the appproxy-router feature:

appproxy_router:
  enabled: true
  coordinator_url: "http://coordinator:10200"
  api_secret: "${APPPROXY_API_SECRET}"        # X-BackendAI-Token (router → coordinator)
  redis_url: "redis://valkey:6379/4"          # event bus DB (stream role)
  authority: "continuum-router-1"
  hostname: "router.example.com"              # advertised host (the LB, if any)
  traffic_port: 443                            # advertised data-plane port; 0 = use api_port
  tls_advertised: true
  heartbeat_period: "10s"
  reconcile_interval: "15s"
  events_enabled: true                         # Redis Pub/Sub overlay on/off

api_secret and redis_url support ${ENV_VAR} interpolation, consistent with backends[].api_key. There is no frontend_mode knob (it is always router), no jwt_secret, no wildcard_domain, and no aggregation-host list: the single endpoint is the aggregation surface. Unknown keys in this section are a hard parse error (operator-authored YAML, unlike the tolerant wire payloads).

Which section is present and enabled determines the worker mode:

Mutual exclusion. In a binary built with both appproxy-legacy and appproxy-router, setting both appproxy.enabled: true and appproxy_router.enabled: true in the same config is rejected at load with a config validation error. Two workers sharing the same authority would each register and then clobber each other's appproxy-* backends on every reconcile pass.

Two global settings are pinned in ROUTER mode:

• api_keys.mode = blocking: forced at startup before any request is served. The coordinator-supplied key set arrives on the first reconcile; blocking mode prevents unauthenticated requests from passing through the window before that completes.
• api_keys.persistence_file: not used. The key set is rebuilt from the coordinator's router-config snapshot on every restart, so configuring a persistence file has no effect in ROUTER mode.

9.12 Security notes¶

• Coordinator auth. All REST calls send X-BackendAI-Token: <api_secret>; keep the secret in env/secret storage and never log it. The secret must be identical across the AppProxy cluster.
• Data-plane auth. Every inference request must present a model API key (Authorization: Bearer or x-api-key) issued by the Manager. The key gates model visibility and use (§9.7); per-circuit JWTs (deployment access tokens) are not used, so ROUTER mode needs no jwt_secret.
• Key custody (hash-only, SHA-256). No recoverable key material is at rest anywhere: the Manager shows the plaintext once at issuance; the coordinator and the router hold only the hash; the router authenticates by hashing the presented bearer. Key events on the Redis bus are id-only notifications, and only hashes move over the authenticated HTTP hops (§9.8). Run both hops over TLS or a trusted network; never log key values (the router redacts token_hash and api_secret in debug output).
• Client IP allow-lists are per key (allowed_client_ips, individual IPs or CIDRs), enforced by the key gate with 403 on mismatch.

9.13 Limitations¶

• Coordinator support is required. ROUTER mode needs the FrontendMode.ROUTER enum value, slot-skipping registration, the /v2/routers/* manager APIs, the router-config events, and the router-config snapshot endpoint on the Backend.AI side as specified by BEP-1053. All of these coordinator capabilities must be available.
• Revocation propagation bound (two-tier). Explicit key revocation reaches workers via a Pub/Sub notification; a node that misses it converges at its next pull, so the worst-case window is reconcile_interval (?strict=true tightens this to all-live-nodes-applied). RBAC-driven implicit revocation is additionally bounded by the Manager's reconcile interval (§9.8).
• Per-node rate limiting. rate_limit is enforced per node with no shared counter, so under HA the effective cluster ceiling is ≈ rate_limit × live_nodes. It is coarse abuse control, not a global quota.
• WILDCARD/PORT slots. A ROUTER worker registers as frontend_mode = router only; per-deployment slot frontends remain the legacy worker's job (run one or the other per process, §9.11).
• Interactive apps. Not served. The router registers accepted_traffics = [inference] only, so interactive circuits stay on stock workers.
• Health/load reporting. The AppProxy heartbeat is a bare keepalive; the router does not export per-circuit load or health metrics to the coordinator (its own Prometheus metrics remain available on the router itself).

10. Security¶

(Legacy worker; for ROUTER mode see §9.12.)

• Coordinator auth. All REST calls send X-BackendAI-Token: <api_secret>. Keep the secret in env/secret storage; never log it.
• Data-plane auth. Non-public inference circuits require an Authorization: Bearer <jwt> whose decoded id equals the circuit id (HS256, jwt_secret). Public circuits (open_to_public == true) skip it. Use jsonwebtoken with Validation::new(Algorithm::HS256) — never the unverified payload decode that exists elsewhere in the tree.
• Client IP allow-list. Honour allowed_client_ips (comma-separated CIDRs) when present on a circuit.
• Shared secrets. api_secret and jwt_secret must be identical across the whole AppProxy cluster (coordinator + workers).

11. Limitations¶

(Legacy worker; for ROUTER mode see §9.13.)

• Model-name source. The served model name is auto-discovered from each replica's /v1/models; the coordinator does not supply a model name (the integration is continuum-router-only, per decision 2).
• PORT mode. Not supported. It would require dynamic per-port listeners; the router registers as a wildcard worker only.
• Interactive apps. Not served. The router registers accepted_traffics = [inference] only, so interactive circuits stay on stock workers.
• Health/load reporting. The AppProxy heartbeat is a bare keepalive; the router does not export per-circuit load or health metrics to the coordinator.